Leveraging Code Coverage Data to Improve Test Suite Efficiency and Effectiveness

摘要

During each release of Visual Studio, substantial time and resources are expended in test case development, execution and verification. Thousands of new tests are added to existing test suites without any kind of review regarding their unique contribution to test suite effectiveness or impact on test suite efficiency. In the past, such unbridled growth in test collateral was sustainable without significantly impacting product release, offset by simply increasing machine and staff resources. With the growing number of test configurations in which these tests need to be run, this is no longer feasible - it is time to clean up!rnIn this paper, we describe how we leverage existing code coverage data, together with reduction techniques, to help each test team analyze its test suite and guide them in improving its effectiveness and efficiency. The analysis focuses on identifying groups of tests cases given specific tactical goals - for example, increasing current test suite stability and reliability, assisting with test migrations, reducing test suite execution time and reducing test suite redundancy. The guidance focuses on a set of best practices that teams can adopt to achieve those goals. The paper reflects on some of the benefits and challenges we faced as part of this case study. It also outlines the tools that were developed to conduct the analysis and support the best practices. We use examples and data taken from the case study to illustrate and emphasize key points.