The Impact of Spelling Correction and Task Context on Short Answer Assessment for Intelligent Tutoring Systems

摘要

This paper explores Short Answer Assess-ment (SAA) for the purpose of giving au-tomatic meaning-oriented feedback in the context of a language tutoring system. In order to investigate the performance of standard SAA approaches on student re-sponses arising in real-life foreign lan-guage teaching, we experimented with two different factors: 1) the incorporation of spelling normalization in the form of a task-dependent noisy channel model spell checker (Brill and Moore, 2000) and 2) training schemes, where we explored task-and item-based splits in addition to stan-dard tenfold cross-validation. For evaluation purposes, we compiled a data set of 3,829 student answers across different comprehension task types col-lected in a German school setting with the English tutoring system FeedBook (Rudzewitz et al., 2017; Ziai et al., 2018) and had an expert score the answers with respect to appropriateness (correct vs. incorrect). Overall, results place the normalization-enhanced SAA system ahead of the standard version and a strong baseline derived from standard text sim-ilarity measures. Additionally, we ana-lyze task-specific SAA performance and outline where further research could make progress.