Following the works of Carletta (1996) and Artstein and Poesio (2008), there is an increasing consensus within the field that in order to properly gauge the reliability of an annotation effort, chance-corrected measures of inter-annotator agreement should be used. With this in mind, it is striking that virtually all evaluations of syntactic annotation efforts use uncor-rected parser evaluation metrics such as bracket F_1 (for phrase structure) and accuracy scores (for dependencies). In this work we present a chance-corrected metric based on Krippendorff's α, adapted to the structure of syntactic annotations and applicable both to phrase structure and dependency annotation without any modifications. To evaluate our metric we first present a number of synthetic experiments to better control the sources of noise and gauge the metric's responses, before finally contrasting the behaviour of our chance-corrected metric with that of un-corrected parser evaluation metrics on real corpora.
展开▼
机译：继Carletta（1996）和Artstein and Poesio（2008）的工作之后，在该领域内越来越多的共识是，为了正确评估注释工作的可靠性，应该使用机会校正的注释者之间协议的度量。考虑到这一点，令人惊讶的是，几乎所有语法注释工作的评估都使用未经校正的解析器评估指标，例如括号F_1（针对短语结构）和准确性得分（针对依存关系）。在这项工作中，我们提出了一种基于Krippendorffα的机会校正量度，该量度适用于句法注释的结构，并且适用于短语结构和依存关系注释，而无需进行任何修改。为了评估指标，我们首先提出了一些综合实验，以更好地控制噪声源并评估指标的响应，然后才将机会校正指标与未经校正的解析器评估指标在真实语料库上的行为进行对比。
展开▼
