On Optimal Uniform Concentration Inequalities for Discrete Entropies in the High-dimensional Setting
share
We prove an exponential decay concentration inequality to bound the tail probability of the difference between the log-likelihood of discrete random variables and the negative entropy. The concentration bound we derive holds uniformly over all parameter values. The new result improves the convergence rate in an earlier result of Zhao (2020), from (K^2log K)/n=o(1) to (log K)^2/n=o(1), where n is the sample size and K is the number of possible values of the discrete variable. We further prove that the rate (log K)^2/n=o(1) is optimal. The results are extended to misspecified log-likelihoods for grouped random variables.
READ FULL TEXT