Looking Under the Hood : Tools for Diagnosing your Question Answering Engine

摘要

In this paper we analyze two question answering tasks : the TREC-8 questionanswering task and a set of reading comprehension exams. First, we show thatQ/A systems perform better when there are multiple answer opportunities perquestion. Next, we analyze common approaches to two subproblems: term overlapfor answer sentence identification, and answer typing for short answerextraction. We present general tools for analyzing the strengths andlimitations of techniques for these subproblems. Our results quantify thelimitations of both term overlap and answer typing to distinguish betweencompeting answer candidates.