Three Information-Theoretical Methods to Estimate a Random Variable

摘要

Three information-theoretical methods to estimate a continuous univariate distribution are proposed for estimation when the distribution type is uncertain, when data are scarce, or when extremes are important. The first is a new version of Jaynes' MaxEnt Method. The second, minimizing Shannon's information measure, yields a minimally informative estimate. The third analysis produces a pair of distributions that together minimize the relative entropy (=cross-entropy) satisfying the principle of least information. Thus, one distribution (p) provides minimal information among all chosen candidates about these events, while the other (q) minimizes the information among all distributions that satisfies the sample rule, which is an essential constraint.