Evaluating Retrieval Performance for Japanese Question Answering: What Are Best Passages?

摘要

Question Answering (QA) has recently received attention from the information retrieval, information extraction, machine learning and natural language processing communities. While traditional Information Retrieval (IR) systems return a list of documents, recent QA systems are tackling the problem of returning short, exact answers in response to open-domain, fact-based questions. TREC started the English QA track at TREC-8 (though systems were to return text snippets instead of exact answers up to TREC-10), and NTCIR started the Japanese QA track at NTCIR-3. The popular approach to QA is the combination of passage retrieval and information extraction. Passage retrieval is used for selecting texts that match the terms extracted from the input question, and information extraction is used for extracting candidate answers from the texts. An important question here is how to define a passage: Long passages (e.g. whole documents) may introduce much noise at the answer selection stage, whereas using short passages (e.g. a few sentences) may imply failure to retrieve texts that contain good answers. How a passage should be defined depends primarily on how the search terms extracted from the question are distributed over each document. At NTCIR-3 QAC1 (Question Answering Challenge 1), a collection of Japanese newspaper articles was used as the knowledge source. Many participants treated each paragraph as a passage, as paragraph boundaries were explicitly given in the newspaper CD-ROM data. This paper questions this popular approach by automatically generating a document retrieval test collection from the QAC1 Question Answering test collection and comparing retrieval performances of five different passage types.