Quantifying Multivariate Classification Performance: The Problem of Overfitting

摘要

We have been studying the use of spectral imagery to locate targets in spectrally interfering backgrounds. In making performance estimates for various sensors it has become evident that some calculations are unreliable because of overflying. Hence, we began a thorough study of the problem of overfitting in multivariate classification. In this paper we present some model based results describing the problem. From the model we know the ideal covariance matrix, the ideal discriminant vector, and the ideal classification performance. We then investigate how experimental conditions such as noise, number of bands, and number of samples cause discrepancies from the ideal results. We also suggest ways to discover and alleviate overfitting.