Efficiency Metrics and Test Case Design for Test Automation

摘要

In this paper, we present our test automation work applied on national e-health portal for residents in Norway which has over million monthly visits. The focus of the work is threefold: delegating automation tasks and increasing reusability of test artifacts; metrics for estimating efficiency when creating test artifacts and designing robust automated test cases. Delegating (part of) test automation tasks from technical specialist (e.g. programmer - expensive resource) to non-technical specialist (e.g. domain expert, functional tester) is carried out by transforming low level test artifacts into high level test artifacts. Such transformations not only reduce dependency on specialists with coding skills but also enables involving more stakeholders with domain knowledge into test automation. Furthermore, we propose simple metrics which are useful for estimating efficiency during such transformations. Examples of the new metrics are implementation creation efficiency and test creation efficiency. We describe how we design automated test cases in order to reduce the number of false positives and minimize code duplication in the presence of test data challenge (i.e. using same test data both for manual and automated testing). We have been using our test automation solution for over three years. We successfully applied test automation on 2 out of 6 Scrum teams in Helsenorge. In total there are over 120 automated test cases with over 600 iterations (as of today).