Learning to Filter Documents for Information Extraction Using Rapid Annotation

摘要

Corpus-driven approaches to information extraction from documents face problems of relevance determination, namely determining which documents are of requisite type, structure, and content for a specified query and context. In this paper, we discuss the problem of learning to filter documents crawled from the web with respect to such relevance criteria, and in particular how to annotate document corpora for supervised classification learning approaches to this problem. For context, we describe a system aimed at extracting experimental data from scientific publications, with the long-term goal of extracting procedural information from relevant sections on experimental methodology. We consider motivating use cases for our learning filter, using the documents passed by the filter: marking up sections (or passages); capturing entities and relationships; and explaining to a domain expert why a document is relevant. These distinct use cases make the annotation task multi-faceted. Our approach focuses on speeding up annotation in learning to filter while minimizing loss of precision or recall on the learning task, using a reconfigurable user interface. We develop such an interface, report on its use in tandem with classification on a real extraction task, and discuss extensions of this work to visual scene filtering and annotation.