Squared-loss Mutual Information Regularization

摘要

The information maximization principle is a useful alternative to the low-density separation principle and prefers probabilistic classifiers that maximize the mutual information (MI) between data and labels. In this paper, we propose an approach for semi-supervised learning called squared-loss mutual information (SMI) regularization, which replaces MI with a novel information measure SMI. SMI regularization is the first framework that can offer all these four properties to algorithms: analytical solution, out-of-sample and multi-class classification, and probabilistic output. As an information-theoretic framework, it is directly related to a manifold regularization, and results in learning algorithms with data-dependent risk bounds. Experiments demonstrate that SMI regularization compares favorably with existing approaches of information-theoretic regularization.