DERIVATION OF MIXTURE DISTRIBUTIONS AND WEIGHTED LIKELIHOOD FUNCTION AS MINIMIZERS OF KL-DIVERGENCE SUBJECT TO CONSTRAINTS

摘要

In this article, mixture distributions and weighted likelihoods are derivedwithin an information-theoretic framework and shown to be closely related.This surprising relationship obtains in spite of the arithmetic form of the former andthe geometric form of the latter. Mixture distributions are shown to be optima thatminimize the entropy loss under certain constraints. The same framework implies theweighted likelihood when the distributions in the mixture are unknown and informationfrom independent samples generated by them have to be used instead. Thusthe likelihood weights trade bias for precision and yield inferential procedures suchas estimates that can be more reliable than their classical counterparts.