How Are You Going to Test All Those Configurations?

摘要

Whether you are testing a traditional application such as a word processor, or you are testing web-based software, the prospect of all the possible configurations that could be tested is a nightmare. We ran head-on into this issue when we started producing web installable updates to our products on an every-other-month basis. We had too many combinations of OSes, browsers, licensing methods, product combinations, and other factors to try in the time available. This paper describes techniques that we evolved which help us expand the set of configurations we test and increase our confidence that our products will function correctly on them. First, we refined our software development process to build the current development products and produce an installable CD image every day. Second, the test group adopted a process involving the establishment of a set of known clean baseline systems for the different configurations we want to test and the use of drive imaging software to quickly restore one of these configurations to a test system. Third, we created a table of the configuration options we use to assign unique testing configurations to each test engineer each time they restore a clean system. The combination of these techniques has allowed our project teams to increase configuration test coverage and reduce the number of configuration problems that make it to our customers.