An Automation and Analysis Framework for Testing Multi-tiered Applications

摘要

In the performance testing of multi-tiered applications, many experiments need to be run with various configurations of parameters where measurements, data collections, analysis and debugging need to be performed. The complexity is further increased by the need to handle multiple layers of software components, including operating systems, virtual machines, databases, and application servers, as well as different hardware platforms. The combinations of different software and hardware platforms under testing and the intense tuning work for each combination unavoidably leads to a very demanding number of quality assurance (QA) performance testing tasks. If daily and weekly software changes are introduced to performance testing, only the most sophisticated QA lab may have enough resources to handle the load - by having sufficient resources and machines to setup for all combinations of configurations that need to be tested. To make it even worse, performance testing often requires many runs with the same set of run-time parameters to assess the performance of the software system. To address the constraints caused by the limited amount of resources, we developed a light-weight approach to automate the performance testing experiments, which can change run-time parameters and schedule runs. This dramatically increases the productivity of the QA engineers. The time saving is about 2 hours per day per engineer after the approach has been implemented versus 8 hours per day per engineer before the change. In addition, the possibility of running wrong experiments (e.g., due to runtime parameter changes) has been greatly reduced. Therefore, the actual productivity gain is bigger than what is measured by savings in time only. Thus, a more than four fold productivity improvement was observed. In this paper we share our experience in automation and analysis framework for performance testing. Our approach to automation and analysis is mostly based on composing components from free tools. We will highlight how we build the tools, what problems we face and what lessons we learned.