High-yield (fault-directed) approach to test suite evaluation.

摘要

Given a test suite T designed to test a program P, there are at least three attributes of T, which can be used to assess T's effectiveness. The first attribute is its Completeness measured by Code Coverage when P is executed on all the tests in T. The second attribute is Adequacy or Fault Coverage of the test suite T, which is the ability of the test suite T to exercise P so that for every fault (according to some fault model) in P there is at least one test case in T that will detect it. The third attribute is Requirements Coverage, which checks for every requirement R whether there is at least one test case in T that will demonstrate the conformance of the program P to the function described by R.; In this thesis we will evaluate the effectiveness of an industrial test suite by measuring the first attribute and estimating the second. In this case T is a Java conformance test suite, and P is a Java compiler. We show that basic coverage measurement techniques and a simple fault model can be used to obtain useful preliminary data early in the testing phase of the software development life cycle. This was the case in our case study. The technique is low-cost and allows the enhancement of conformance test suite for languages, protocols, or any standard specification.