Preserving a combat commander's moral agency: The Vincennes Incident as a Chinese Room

摘要

We argue that a command and control system can undermine a commander's moral agency if it causes him/her to process information in a purely syntactic manner, or if it precludes him/her from ascertaining the truth of that information. Our case is based on the resemblance between a commander's circumstances and the protagonist in Searle's Chinese Room, together with a careful reading of Aristotle's notions of 'compulsory' and 'ignorance'. We further substantiate our case by considering the Vincennes Incident, when the crew of a warship mistakenly shot down a civilian airliner. To support a combat commander's moral agency, designers should strive for systems that help commanders and command teams to think and manipulate information at the level of meaning. 'Down conversions' of information from meaning to symbols must be adequately recovered by 'up conversions', and commanders must be able to check that their sensors are working and are being used correctly. Meanwhile ethicists should establish a mechanism that tracks the potential moral implications of choices in a system's design and intended operation. Finally we highlight a gap in normative ethics, in that we have ways to deny moral agency, but not to affirm it.