Beware of External Validation! - A Comparative Study of Several Validation Techniques used in QSAR Modelling

摘要

Background: Proper validation is an important aspect of QSAR modelling. Externalvalidation is one of the widely used validation methods in QSAR where the model is built on a subsetof the data and validated on the rest of the samples. However, its effectiveness for datasets with asmall number of samples but a large number of predictors remains suspect.Objective: Calculating hundreds or thousands of molecular descriptors using currently availablesoftware has become the norm in QSAR research, owing to computational advances in the past fewdecades. Thus, for n chemical compounds and p descriptors calculated for each molecule, the typicalchemometric dataset today has a high value of p but small n (i.e. n p). Motivated by the evidenceof inadequacies of external validation in estimating the true predictive capability of a statistical modelin recent literature, this paper performs an extensive and comparative study of this method with severalother validation techniques.Methodology: We compared four validation methods: Leave-one-out, K-fold, external and multi-splitvalidation, using statistical models built using the LASSO regression, which simultaneously performsvariable selection and modelling. We used 300 simulated datasets and one real dataset of 95congeneric amine mutagens for this evaluation.Results: External validation metrics have high variation among different random splits of the data,hence are not recommended for predictive QSAR models. LOO has the overall best performanceamong all validation methods applied in our scenario.Conclusion: Results from external validation are too unstable for the datasets we analyzed. Based onour findings, we recommend using the LOO procedure for validating QSAR predictive models built onhigh-dimensional small-sample data.