Evaluating testing methods by delivered reliability [software]

摘要

There are two main goals in testing software: (1) to achievenadequate quality (debug testing), where the objective is to probe thensoftware for defects so that these can be removed, and (2) to assessnexisting quality (operational testing), where the objective is to gainnconfidence that the software is reliable. Debug methods tend to ignorenrandom selection of test data from an operational profile, while fornoperational methods this selection is all-important. Debug methods arenthought to be good at uncovering defects so that these can be repaired,nbut having done so they do not provide a technically defensiblenassessment of the reliability that results. On the other hand,noperational methods provide accurate assessment, but may not be asnuseful for achieving reliability. This paper examines the relationshipnbetween the two testing goals, using a probabilistic analysis. We definensimple models of programs and their testing, and try to answer thenquestion of how to attain program reliability: is it better to test bynprobing for defects as in debug testing, or to assess reliabilityndirectly as in operational testing? Testing methods are compared in anmodel where program failures are detected and the software changed toneliminate them. The “better” method delivers highernreliability after all test failures have been eliminated. Special casesnare exhibited in which each kind of testing is superior. An analysis ofnthe distribution of the delivered reliability indicates that even simplenmodels have unusual statistical properties, suggesting caution inninterpreting theoretical comparisons