Automated Techniques for Improving the Quality of Existing Test Suites

摘要

Testing is playing a crucial and fundamental role in modern software development. Although software tests are conceptually simple---they are composed of two primary parts: inputs that are used to execute the program under test and an oracle that is used to verify that the execution induced by the inputs produces the expected results---they are often difficult to write in practice. The software engineering research community provided many techniques that can help developers determine whether they have written effective and efficient tests, including various coverage metrics which have been widely adopted. While they have been proven successful in practice, many follow-up studies show that there is still a lot to improve for the test quality measurements, including false negatives in alarming the lack of tests and constructive suggestions for improvements.;This dissertation focuses on improving the quality of existing test suites based on interpretations on test inputs and test oracles. If a test oracle checks the values which the test developers did not or can not control, it would make the test brittle. I developed a novel technique based on dynamic tainting which can identify the values that can make tests brittle. An empirical study on real-world applications shows that the technique can reveal brittle assertions and the values that cause the brittleness within reasonable cost. I also developed two techniques that can identify insufficiently tested code by interpreting traditional coverage information. One is a new approach based on the concepts of direct coverage and indirect coverage. The other is a new approach to discover incidentally tested code. Both techniques have shown efficiency and effectiveness in the empirical studies on real-world applications.