Performance Evaluation of Artificial Intelligence Systems

摘要

The major premise is that the DoD is moving towards the large-scale use of Artificial Intelligence Systems (AISs) as tools for supporting decision making and information analysis in military domains. It becomes increasingly important, therefore, that we develop mechanisms both for assessing the impact that these systems have upon the ability of the individual decision makers and analysts to carry out their assignments, and for assessing the organizational impact that this would have. This is what we choose to call the problem of Performance Evaluation. This report describes the development of a methodology for performance evaluation. This approach provides for both multiple perspectives on the problem and multiple evaluation techniques. The central idea in our methodology is that of an evaluation frame. This is the main organizational element in this representation of performance knowledge. To validate the methodology, a number of example evaluations are performed of expert systems building tools and decision aids, also using the results to help us the define a prototype evaluation environment. The evaluations and the design are reported in full. This proposed methodology has a high likelihood of success.