Summarization Evaluation meets Short-Answer Grading

摘要

Summarization Evaluation and Short-Answer Grading share the challenge of automatically evaluating content quality. Therefore, we explore the use of ROUGE, a well-known Summarization Evaluation method, for Short-Answer Grading. We find a reliable ROUGE parametrization that is robust across corpora and languages and produces scores that are significantly cor-related with human short-answer grades. ROUGE adds no information to Short-Answer Grading NLP-based machine learn-ing features in a by-corpus evaluation. However, on a question-by-question basis, we find that the ROUGE Recall score may outperform standard NLP features. We therefore suggest to use ROUGE within a framework for per-question feature se-lection or as a reliable and reproducible baseline for SAG.