An empirical comparison of validation methods for software prediction models

摘要

Model validation methods (e.g., k-fold cross-validation) use historical data to predicthow well an estimation technique (e.g., random forest) performs on the current(or future) data. Studies in the contexts of software development effort estimation(SDEE) and software fault prediction (SFP) have used and investigated differentmodel validation methods. However, no conclusive indications to suggest whichmodel validation method has a major impact on the prediction accuracy and stabilityof estimation techniques. Some studies have investigated model validation methodsspecific to data about either SDEE or SFP. To the best of our knowledge, there is nostudy in the literature, which has employed different validation methods both withSDEE and SFP data. The aim of this paper is to consider different methods (10) fromthe family of cross-validation (CV) and bootstrap validation methods to identifywhich one contributes to obtaining a better prediction accuracy for both types ofdata. We also evaluate which model validation methods allow the estimationtechniques to provide stable performances (i.e., with lower variance). To this aim, wepresent an empirical study involving six datasets from the domain of SDEE and sixdatasets from the SFP domain. The results reveal that repeated 10-fold CV withSDEE and optimistic boot with SFP data are the model validation methods thatprovide a better prediction accuracy in a greater number of experiments than theother model validation methods. Furthermore, a model validation method canimprove the prediction accuracy up to 60% with SDEE data and up to 36% whenemploying SFP data. The analysis also reveals that repeated fivefold CV producesmore stable performances when the experiments are repeated on the same data.