Minimax Estimation of Discrete Distributions Under $ell _{1}$ Loss

摘要

We consider the problem of discrete distribution estimation under loss. We provide tight upper and lower bounds on the maximum risk of the empirical distribution (the maximum likelihood estimator), and the minimax risk in regimes where the support size may grow with the number of observations . We show that among distributions with bounded entropy , the asymptotic maximum risk for the empirical distribution is , while the asymptotic minimax risk is . Moreover, we show that a hard-thresholding estimator oblivious to the unknown upper bound , is essentially minimax. However, if we constrain the estimates to lie in the simplex of probability distributions, then the asymptotic minimax risk is again . We draw connections between our work and the literature on density estimation, entropy estimation, total variation distance ( divergence) estimation, joint distribution estimation in stochastic processes, normal mean estimation, and adaptive estimation.