Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy.

摘要

It is proven that, in a well-defined sense, Jaynes's principle of maximum entropy and Kullback's principle of minimum cross-entropy (minimum directed divergence) provide uniquely correct, general methods of inductive inference when new information is given in the form of expected values. Previous justifications rely heavily on intuitive arguments and on the properties of entropy and cross-entropy as information measures. The approach assumes that reasonable methods of inductive inference should lead to consistent results whenever there are different ways of taking the same information into account --- for example, in different coordinate systems. This requirement is formalized as four consistency axioms stated in terms of an abstract information operator; the axioms make no reference to information measures. This result is established both directly and as a special case (uniform priors) of an analogous, more general result for the principle of minimum cross-entropy. Results are obtained both for continuous probability densities and for discrete distributions.