Whither Written Language Evaluation?

摘要

Common evaluations have grown to be a major component of all the ARPA Human Language Technology programs. In the written language community, the largest evaluation program has been the series of Message Understanding Conferences, which began in 1987. These evaluations have focussed on the task of analyzing text and automatically filling templates describing certain classes of events. These conferences have certainly been a major impetus in the development of systems for performing such "information extraction" tasks, and thus in demonstrating the potential practical value of some of the written language processing technology.rnThere have been a number of concerns expressed, however, about the trend of these evaluations. First, these evaluations - and particularly the most recent, MUC-5 - have consumed large amounts of time, and in particular time spent learning and encoding detailed information about the domain, rather than learning about how to process language in general. Second, there has been a focus on technologies which are effective for this task but may not be effective for other, "language understanding" tasks.rnIn response to these concerns, a group of ARPA contractors and Government representatives met on December 2-4 in San Diego to plan for the next written language evaluation conference (MUC-6). This paper is a report on the conclusions of this meeting and some of the electronic interchanges which followed.