In order to evaluate software performance and find regressions, manydevelopers use automated performance tests. However, the test results oftencontain a certain amount of noise that is not caused by actual performancechanges in the programs. They are instead caused by external factors likeoperating system decisions or unexpected non-determinisms inside theprograms. This makes interpreting the test results hard since results thatdiffer from previous results cannot easily be attributed to either genuinechanges or noise.In this thesis we use Mozilla Firefox as an example to try to find thecauses for this performance variance, develop ways to reduce the noise andpresent a statistical technique that makes identifying genuine performancechanges more reliable.Our results show that a significant amount of noise is caused by memoryrandomization and other external factors, that there is variance in Firefoxinternals that does not seem to be correlated with test result variance, andthat our suggested statistical forecasting technique can give more reliabledetection of genuine performance changes than the one currently in use byMozilla.
展开▼
