Issues in the conduct of comparisons among human performance models and live human performance in complex simulated task environments

摘要

The Air Force Research Laboratory has sponsored the Agent-based Modeling and Behavior Representation (AMBR) Project to advance the state of the art in human performance modeling in general and the state of practice in cost-effective computer generated forces more specifically (Pew & Mavor, 1998). To advance human performance modeling they have created an opportunity for multiple developers to create different models of the same human operator activity and to compare the results both from model to model and with human participants performing the same task. As you will learn in a later talk, in the course of this project these model comparisons will be conducted three times, each time using different, hopefully more demanding, human modeling requirements. The first comparison is now complete and the results were reported at the International Ergonomics Society Meeting in July 2000. The project is important because it is rare to have the opportunity to validate human performance models, and even rarer to be able to compare and contrast the results of multiple model developers who use different model architectures and draw their models from different theoretical perspectives. BBN has been assigned the responsibility of serving as moderator for these comparisons. In that capacity, for the first round, we developed a simulated, but much simplified, air traffic control environment and task scenarios that emphasized multiple task management, task priority setting and attention management. Using DOMAR (Freeman, 1997), an agent-based simulation environment, we have the capability to interchangeably "plug in" a human operator model or a live human participant. This capability was used to collect human performance data exercising scenarios that were identical to those the model developers were required to exercise. Together with Michael Young, the project initiator, we developed the specifications the model developers were to respond to. We collected the human performance data supplied to the developers for developing their models and the data that was used for model evaluation. Together with the developers, we conducted the experimental comparisons among models, analyzed and reported the results. Four independent development teams participated. In the course of completing these assignments we have had to address a number of challenging issues. This paper describes a sample of these issues and what we have done to resolve them. I make no claim that our solutions are the best ones. In some cases they were driven by practical considerations alone. I hope the paper will stimulate others to think about how they should be resolved.