Normalization in Context: Inter-Annotator Agreement for Meaning-Based Target Hypothesis Annotation

摘要

We explore the contribution of explicit task contexts in the annotation of word-level and sentence-level normalizations for learner language. We present the annotation schemes and tools used to annotate both word- and sentence-level target hypotheses given an explicit task context for the Corpus of Reading Exercises in German (Ott et al., 2012) and discuss a range of inter-annotator agreement measures appropriate for evaluating target hypothesis and error annotation. For learner answers to reading comprehension questions, we find that both the amount of task context and the correctness of the learner answer influence the inter-annotator agreement for word-level normalizations. For sentence-level normalizations, the teachers' detailed assessments of the learner answer meaning provided in the corpus give indications of the difficulty of the target hypothesis annotation task. We provide a thorough evaluation of inter-annotator agreement for multiple aspects of meaning-based target hypothesis annotation in context and explore measures beyond inter-annotator agreement that can potentially be used to evaluate the quality of normalization annotation.