What Could Safety Research Contribute to Technology Design?

摘要

There is already evidence of several mishaps, accidents, and even major catastrophes in which AI technologies have played a part. Among them are traditional types of organizational accidents but also complex misadventures where the effects of the use of AI technology intertwine with human action in the context of intricate socio-technical settings. The origins of problems range from making errors during a planned course of action to accidents due to unintended and unpredictable outcomes of the use of AI. A variety of ethical principles for the use and design of AI have been formulated to remedy the ills. The article sums up the main concepts of the AI ethical codes and discusses their role in attempts to prevent AI-induced problems. According to recent empirical studies, the impact of ethics principles is weak in terms of real-life AI design and use. As an alternative, the article suggests the findings of safety research as a source for a practice-oriented approach to tackle the problems. A century-long study of safety provides both conceptual models and practical tools for the prevention of accidents and creating the means to improve safety. The article introduces the main findings of this line of research and suggests how the Normal Accident Theory, High Reliability Organizations studies, and Just Culture approach could contribute to a safer AI design.