Identifying and Prioritizing Chaos Experiments by Using Established Risk Analysis Techniques

摘要

The prevalence of microservice architectures and container orchestration technologies increases the complexity of assessing such systems’ resilience. Chaos engineering is an emerging approach for resilience assessment by testing hypotheses after intentionally injecting faults into a distributed system and observing customer- and business-affecting metrics. As the number of potential risks within a complex system is high, the identification and prioritization of effective and efficient chaos experiments are non-trivial. In the scope of an industrial case study, this work investigates means to identify and prioritize chaos experiments by using established risk analysis techniques known from engineering safety-critical systems, namely i) Fault Tree Analysis, ii) Failure Mode and Effects Analysis, iii) and Computer Hazard and Operability Study. We conducted semi-structured interviews to elicit architectural information and resilience requirements of the case study system. The extracted knowledge was leveraged during the application of the risk analysis techniques. A subset of the identified and prioritized risks was used to create and execute chaos experiments. The risk analysis resulted in over 100 findings and revealed that the system is rather fragile as it comprises a high amount of single points of failure. The chaos experiments revealed further weaknesses for formerly unknown system behavior.