Predicting post-release defects using pre-release field testing results

摘要

Field testing is commonly used to detect faults after the in-house (e.g., alpha) testing of an application is completed. In the field testing, the application is instrumented and used under normal conditions. The occurrences of failures are reported. Developers can analyze and fix the reported failures before the application is released to the market. In the current practice, the Mean Time Between Failures (MTBF) and the Average usage Time (AVT) are metrics that are frequently used to gauge the reliability of the application. However, MTBF and AVT cannot capture the whole pattern of failure occurrences in the field testing of an application. In this paper, we propose three metrics that capture three additional patterns of failure occurrences: the average length of usage time before the occurrence of the first failure, the spread of failures to the majority of users, and the daily rates of failures. In our case study, we use data derived from the pre-release field testing of 18 versions of a large enterprise software for mobile applications to predict the number of post-release defects for up to two years in advance. We demonstrate that the three metrics complement the traditional MTBF and AVT metrics. The proposed metrics can predict the number of post-release defects in a shorter time frame than MTBF and AVT.