Numerous chemical data sets have become available for quantitative structure–activity relationship (QSAR) modeling studies. However, the quality of different data sources may be different based on the nature of experimental protocols. Therefore, potential experimental errors in the modeling sets may lead to the development of poor QSAR models and further affect the predictions of new compounds. In this study, we explored the relationship between the ratio of questionable data in the modeling sets, which was obtained by simulating experimental errors, and the QSAR modeling performance. To this end, we used eight data sets (four continuous endpoints and four categorical endpoints) that have been extensively curated both in-house and by our collaborators to create over 1800 various QSAR models. Each data set was duplicated to create several new modeling sets with different ratios of simulated experimental errors (i.e., randomizing the activities of part of the compounds) in the modeling process. A fivefold cross-validation process was used to evaluate the modeling performance, which deteriorateswhen the ratio of experimental errors increases. All of the resultingmodels were also used to predict external sets of new compounds, whichwere excluded at the beginning of the modeling process. The modelingresults showed that the compounds with relatively large predictionerrors in cross-validation processes are likely to be those with simulatedexperimental errors. However, after removing a certain number of compoundswith large prediction errors in the cross-validation process, theexternal predictions of new compounds did not show improvement. Ourconclusion is that the QSAR predictions, especially consensus predictions,can identify compounds with potential experimental errors. But removingthose compounds by the cross-validation procedure is not a reasonablemeans to improve model predictivity due to overfitting.
展开▼
