Intelligence Analysis Problems with Open-Ended Solution Spaces: Testing and Evaluation of Systems

摘要

The dire importance of improving the ability to make decisions and predictions in open-ended circumstances was brought home by the disaster September 11, 2001. Looking back, numerous factors have since been identified that appear to be indicative of the events that were to come. It is possible that these dues originally were missed due to faulty processes or constraints on resources. Laudable efforts have been made to fix the processes and to acquire new systems to better support such open-ended decision-making. However, testing and evaluating these new systems is difficult because of two characteristics that define open-ended decision situations: (1) non-uniqueness of situation models that therefore yields non-unique possible predictions from indicators; and (2) subjectivity in evaluating indicators. This paper examines these problems and provides suggestions on how previously used procedures applied to evaluation of other systems with open-ended characteristics could be adapted to intelligence analysis systems.