System to uncover root cause of non-deterministic (flaky) tests

摘要

A system and method are disclosed for determining the root cause of non-deterministic tests. An exemplary system may receive a non-deterministic test and run the non-deterministic test with call trace collection enabled. For each trace, the system may determine whether the trace was associated with a passing or failing test. Each trace may be labeled according to the trace's association. Then, dynamic call trees may be constructed, one for the traces associated with passing tests and one for traces associated with failing tests. The dynamic call trees may be compared to determine subtree patterns that occur in one tree, but not the other. The subtree patterns may then be ranked based on the number of times the subtrees occur in the traces in order to determine the root cause of the non-deterministic test.