Model-Based Analysis of Semi-Automated Data Discovery and Entry Using Automated Content Extraction

摘要

Content extraction systems can automatically extract entities and relations from raw text and use the information to populate knowledge bases, potentially eliminating the need for manual data discovery and entry. Unfortunately, content extraction is not sufficiently accurate for end-users who require high trust in the information uploaded to their databases, creating a need for human validation and correction of extracted content. In this paper, we examine content extraction errors and explore their influence on a prototype semi-automated system that allows a human reviewer to correct and validate extracted information before uploading it, focusing on the identification and correction of precision errors. We applied content extraction to six different corpora and used a Goals, Operators, Methods, and Selection rules Language (GOMSL) model to simulate the activities of a human using the prototype system to review extraction results, correct precision errors, ignore spurious instances, and validate information. We compared the simulated task completion rate of the semi-automated system model with that of a second GOMSL model that simulates the steps required for finding and entering information manually. Results quantify the efficiency advantage of the semi-automated workflow and illustrate the value of employing multidisciplinary quantitative methods to calculate system-level measures of technology utility.