首页> 外文OA文献 >Crowdsourcing ground truth for Question Answering using CrowdTruth
【2h】

Crowdsourcing ground truth for Question Answering using CrowdTruth

机译:使用CrowdTruth进行问题解答的众包基础事实

摘要

Gathering training and evaluation data for open domain tasks, such as general question answering, is a challenging task. Typically, ground truth data is provided by human expert annotators, however, in an open domain experts are difficult to define. Moreover, the overall process for annotating examples can be lengthy and expensive. Naturally, crowdsourcing has become a mainstream approach for filling this gap, i.e. gathering human interpretation data. However, similar to the traditional expert annotation tasks, most of those methods use majority voting to measure the quality of the annotations and thus aim at identifying a single right answer for each example, despite the fact that many annotation tasks can have multiple interpretations, which results in multiple correct answers to the same question. We present a crowdsourcing-based approach for efficiently gathering ground truth data called CrowdTruth, where disagreement-based metrics are used to harness the multitude of human interpretation and measure the quality of the resulting ground truth. We exemplify our approach in two semantic interpretation use cases for answering questions.
机译:为开放领域任务(例如一般问题解答)收集培训和评估数据是一项具有挑战性的任务。通常,地面真相数据是由人类专家注释者提供的,但是,在开放域中,专家很难定义。而且,用于注释示例的整个过程可能是冗长且昂贵的。自然地,众包已成为填补这一空白(即收集人类解释数据)的主流方法。但是,类似于传统的专家批注任务,尽管许多批注任务可以有多种解释,但大多数方法都使用多数投票来衡量批注的质量,从而旨在为每个示例识别单个正确答案。导致对同一问题的多个正确答案。我们提出了一种基于众包的方法,可以有效地收集称为CrowdTruth的地面事实数据,其中基于分歧的指标可用于利用多种人类解释并衡量所产生的地面事实的质量。我们在两个用于解释问题的语义解释用例中举例说明了我们的方法。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利

客服邮箱:kefu@zhangqiaokeyan.com

京公网安备:11010802029741号 ICP备案号:京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有
  • 客服微信

  • 服务号