Why the pilot cannot be blamed: a cautionary note about excessive reliance on technology

摘要

In many human endeavours intelligent automation has taken over some of the tasks traditionally performed by operators, pilots or controllers. The adoption of intelligent protective technology reflects the greater degree of reliability normally ascribed to such systems. Intelligent technology is often credited with saving lives and reducing accidents. This paper looks at the crash of a revolutionary supersonic fighter that resulted from over-reliance on protection technology. The degree of automation of the protection system made it impossible for the pilot to regain control and convince the system that there was a problem. Technology has thus created a new kind of computer-assisted error, where a system designed to make a task safer is actually directly responsible for causing a disaster. Developers thus need to foresee the impact of new technology in its original situational context and consider the implications of wresting control away from the pilot and giving it to the computer.