Applying Selective Revalidation Techniques at Microsoft

摘要

The Internet Explorer (IE) test team faces two key testing challenges: 1. Efficiently and effectively regression testing code changes in the context of a large and complex code base 2. Managing the size and complexity of a large suite of tests while maintaining its effectiveness At the same time, the team also wants to maintain the quality bar across the range of different IE versions, the Windows platforms on which IE runs and its 32-/64-bit binary releases. All of these factors are then also being considered in the context of shorter development test cycles. Our approach to addressing these challenges is to apply so-called selective revalidation techniques, which leverage our existing code coverage data. Regression test selection enables a potentially significant reduction in the number of regression tests that need to be rerun in response to a given code modification. Test set optimization focuses on a mechanism to better prioritize our large test suite and identify potential redundancy. We anticipate that both of these techniques will not only encourage more systematic and efficient testing procedures, but significantly streamline our current test development cycle.While research into selective revalidation techniques has been conducted for thirty years, there have been very few publications that have discussed this topic beyond the academic setting. This paper attempts to address this and showcases the activities conducted in this area within Microsoft. This paper examines specific challenges facing the IE team and outlines its requirements for selecting selection revalidation techniques. We describe how these requirements were met by these techniques and their application to key activities within the IE development test process. We also demonstrate how existing in-house tools have been leveraged, customized and deployed to support the process. Examples, taken from the recent IE7/Vista development lifecycle, are given throughout.