A weighted quadratic asymptotic analysis of cost functions used in classifier design with extensions to finite-size training sets

摘要

An analysis of the impact of cost function on classifier design is presented. The well know asymptotic probabilistic approach, that invokes the law of large numbers, is extended by incorporating a piece-wise weighted quadratic approximation. This allows different cost functions to be compared and better quantifies the impact of the cost function on the resulting classifier design. In this paper we show how the choice of several well known cost functions are related to (1) Bayesian optimality, (2) classifier complexity, and (3) the ability to estimate decision boundaries. This work extends previous work that relates classifier design to approximations of the Bayesian posterior probability of class membership (e.g., "Any Reasonable Cost Function Can be Used for A Posteriori Probability Approximation" by M. Saerens, et al., IEEE Transactions on NN, September 2002). Several cost functions are analyzed in the paper including the L_p norm and the maximum mutual information (MMI) criterion. An interesting example that supports the theoretical analysis is presented. For the example the L_p norm (with p=1.1) was shown to successfully estimate the Bayesian optimal class decision boundary while the MMI and the L_2 criteria did not. In addition, a finite-version of the theory is presented that bridges the gap between asymptotic theory and strictly finite-size training sets.