Concentration Inequalities for the Empirical Distribution
share
We study concentration inequalities for the Kullback--Leibler (KL) divergence between the empirical distribution and the true distribution. Applying a recursion technique, we improve over the method of types bound uniformly in all regimes of sample size n and alphabet size k, and the improvement becomes more significant when k is large. We discuss the applications of our results in obtaining tighter concentration inequalities for L_1 deviations of the empirical distribution from the true distribution, and the difference between concentration around the expectation or zero.
READ FULL TEXT