Minimum Cross-Entropy Pattern Classification and Cluster Analysis

摘要

This paper considers the problem of classifying an input vector of measurements by a nearest-neighbor rule applied to a fixed set of vectors. The fixed vectors are sometimes called characteristic feature vectors, codewords, cluster centers, models, reproductions, etc. The nearest-neighbor rule considered uses a non-Euclidean, information-theoretic distortion measure that is not a metric, but that nevertheless leads to a classification method that is optimal in a well-defined sense and is also computationally attractive. Furthermore, the distortion measure results in a simple method of computing cluster centroids. Our approach is based on cross-entropy minimization (also called minimum discrimination information or minimum directed divergence), and can be viewed as a refinement of a general classification method due to Kullback. The refinement exploits special properties of cross-entropy that hold when the probability densities involved happen to be minimum cross-entropy densities. The approach is a generalization of a recently-developed speech coding technique.