Automation Surprise in the 21st Century: Culture, Collaborative Cognition, Complexity and Legacy Systems

摘要

Automation surprises continue to be a significant safety concern, and the system safety community's search for effective strategies to mitigate them are ongoing. The literature has offered two fundamentally divergent directions, based on different ideas about the nature of cognition and collaboration with automation: a normative individual-cognition model and a sensemaking model based on distributed cognition. In this talk, I will run through the human factors and system safety investigation of an accident that illustrates an undiscovered niche of catastrophic risk-even in an ultra-safe system. A combination of culture, buggy mental models and legacy systems contributed critically in this case, and there was no appropriate training, no written guidance, no documentation, and no likelihood of operational experience that would insulate people from the kind of automation surprise and accident that happened. The sensemaking model suggests that our understanding of the interaction between humans and automation in a case such as the one discussed can be improved by taking into account culture, collaborative cognition, complexity and legacy systems vis-a-vis the operational context, rather than focusing on suboptimal human performance.