Testing for outliers from a mixture distribution when some data are missing

摘要

The problem being considered is that of multivariate outlier testing from a population from which a training sample is available. A new observation is obtained and is tested to determine whether it is from the population of the training sample. Problems of this type arise in a number of applications including nuclear monitoring, biometrics (including fingerprint and handwriting identification), and medical diagnosis. In many cases it is reasonable to model the population of the training sample using a mixture-of-normals model (e.g. when the observations come from a variety of sources or the data are substantially non-normal). A modified likelihood ratio test is considered that is applicable to the case in which: (a) the training data follow a mixture-of-normals distribution, (b) all labels in the training sample are missing, (c) some of the observation vectors in the training sample have missing information, and (d) the number of components in the mixture is unknown. The approach often used in practice when some of the data vectors have missing observations is to perform the test based only on the data vectors with full data. When large amounts of data are missing, use of this strategy may lead to loss of valuable information, especially in the case of small training samples which, for example, is often the case in the nuclear monitoring setting. This paper discusses an alternative procedure that incorporates all n of the data vectors using the expectation-maximization (EM) algorithm to handle the missing data. Simulations and examples are used to compare the use of the EM algorithm on the entire data set with the use of only the complete data vectors.