IMPROVING TEST AUTOMATION BY DETERMINISTIC METHODS IN STATISTICAL TESTING

摘要

Statistical testing is of increasing interest because it allows full test automation – from test generation to evaluation - and hence reduces significantly the human test effort, while allowing a much broader test range. However, automatically generated tests based on a statistical approach have to cope with the "oracle problem" and the "small target problem". The oracle problem represents the fact that software cannot conclude on the correctness of the derived results by itself, while the small target problem consists in the challenge of hitting sporadic, but important test conditions in a large input domain. In the worst case this problem can result in zeroprobability for specific test conditions. Although deterministic testing methods in principle do not suffer from the small target problem, they suffer from an oracle problem as well, which is the problem to know which test cases are needed. In general, the test criteria need to represent all viable questions a test engineer wants to pose on the respective system under test. Therefore deterministic methods are best at pointing out anticipated faults (as far as “anticipated” by a test engineer) while statistical methods can also reveal non-anticipated faults. BSSE has started already more than ten years ago with first activities in statistical testing, and built related tools. Based on the feedback from recent activities an approach has been defined aiming to overcome the weakness of statistical and deterministic testing by making a synthesis of the best of the two worlds. This approach includes automatic test case generation based on the information in the source code (prototype specification and code structure), optimisation by improved test criteria for statistical testing and methods known from deterministic testing,automated test evaluation and comprehensive presentation of results.