Directly Comparing Computer and Human Performance in Language Understanding and Visual Reasoning.

摘要

This paper addresses two of the major areas for the first phase of work on the Artificial Intelligence Measurement System (AIMS) Project: natural language understanding and vision systems. One particular goal was to determine the extent to which AI systems can be described in terms of similar performance by groups of people. To that end, the plan was to develop appropriate evaluation models that would allow for description and assessment of AI programs in a variety of ways. The technical approach was threefold: (1) maps of the problems that had been addressed by AI program were to be created; (2) criteria by which any given programs could be assessed were to be identified--functional specifications, cognitive processing models underlying development, nuts and bolts of design and implementation, etc.; (3) computer implementations were to be described in terms of human performance. The focus of this paper is on the third area. Project efforts in human benchmarking are reviewed with regard to both natural language understanding and vision.