An effective implementation of analytical question answering.

摘要

Analytical question answering (AQA) is the task of finding precise and complete answers to a series of complex, exploratory questions. These questions form a part of an analytical strategy to produce a report addressing a complex information problem. AQA underlies many real life human activities, including intelligence analysis, legal research, news production, etc. In this thesis we have developed a groundbreaking approach to implementing AQA capability in a computer system. Many earlier approaches to automated question answering are not applicable to AQA because they have overwhelmingly focused on isolated fact-recall questions and thus largely ignored the contextual and exploratory aspects of complex questions. In this work we recognize three distinct challenges that researchers face in attempting to build a successful AQA system: the massive, unstructured data sources (including unstructured text), the under-defined domain model and the implicit user information task (the scenario). While the first of these challenges can be addressed using robust but imprecise information retrieval techniques, the other two challenges have until now remained the major stumbling blocks for implementing practical workable AQA technology. In this thesis we present initial solutions to both challenges using (a) data driven semantics for interpreting NL questions and (b) mismatch-driven human-machine dialogue for negotiating the exact scope of the answer. In order to demonstrate viability of our solutions, we have built an end-to-end AQA system. Our system was invited to participate in the ARDA Metrics Challenge Workshop run by NIST in June 2004 which provided an opportunity to perform an objective evaluation with active duty analysts. In addition to traditional metrics of answer accuracy, which are only partial measures of system performance, we were particularly interested in measures of increased efficiency from the user viewpoint as well as user satisfaction including confidence in the final result. The experiments conducted in the course of this work indicate that our approach to analytical QA increases user's efficiency by at least 120% as compared to keyword based search methods (such as Google). Additionally, when using our system, analysts spent 24% less time to produce their reports, while achieving higher report scores in a cross-evaluation.