Model validation software for classification models using repeated partitioning: MVREP.

摘要

The process of assessing the prediction ability of a computational model is called model validation. For models predicting a categorical response, the prediction ability is usually quantified by prediction measures such as sensitivity, specificity, and accuracy. This paper presents a software Model Validation using Repeated Partitioning (MVREP) that implements a computer-intensive, nonparametric approach to model validation, which we call the re-partitioning method. MVREP, developed using the SAS Macro language, repeats the process of randomly partitioning a dataset and subsequently performing standard model validation procedures, such as cross-validation, a large number of times and generates the empirical sampling distributions of prediction measures. The means of the sampling distributions serve as the point estimates of prediction measures of the model. The variances of the sampling distributions provide a direct assessment of variability for the point estimates of prediction measures. An example is presented using a mouse developmental toxicity chemical dataset to illustrate how the software can be used for the assessment of structure-activity relationships models.