Profile analysis techniques for observation-based software testing.

摘要

Observation-based testing is a software-testing paradigm based on the idea of observing the behavior of the program when executed under a variety of test cases. The runtime behavior of a program can be summarized in profiles, which can then be analyzed for a variety of purposes useful for the tester. This dissertation presents techniques for test suite visualization, test case selection and test case prioritization based on profile data and includes extensive experiments on large, real-world applications to compare these techniques with ones from the literature.; Test suite visualization is the application of multivariate visualization techniques to profile data in order to visually study the composition of the test suite and its interaction with the program. Two techniques are examined for this purpose, Correspondence Analysis and Multidimensional Scaling, and a novel algorithm for the latter is presented and studied. Example applications of test suite visualization are provided.; Test case selection is the problem of selecting a small set of tests from a large test suite such that the most defects are revealed when this subset is executed. Test case prioritization is the problem of finding an optimal scheduling of the tests in a test suite so that the number of defects found earlier during testing is maximized. Other researchers have tried to address these problems using profile information, by looking at the amount of code executed by a subset of tests. Dickinson proposed some methods for test-case selection that consider the distribution of the profiles in the profile space by using cluster analysis on the profiles. This work was later extended in conjunction with the author. These methods will be presented in this work, together with novel methods for test case prioritization. Experimental validations and comparisons of all of these methods will be presented, including comparison criteria that were missing from earlier work.; The results suggest that profile analysis is a useful tool for software testers, and that studying the distribution of tests in a profile space can be more beneficial than concentrating on code coverage.