Automated Evaluation of Short Summaries

摘要

Children in elementary school are not only taught to read, but to understand what they are reading. To assess and improve their ability to understand concepts, students are often required to write short summaries of articles. Due to their nature, these documents often include misspelled words, missing punctuation, and erroneous grammatical structure. Evaluating these summaries is a laborious task that not only demands a significant amount of time from professors, but also limits the speed in which students can receive feedback This paper presents a method for evaluating short summaries written by elementary school students. Our experiments show that incorporating semantic similarity/relatedness measures between words benefits the tasks of attribute selection and attribute weighting. We also show that preprocessing steps, such as the correction of misspelled words, are beneficial for the evaluation of short summaries. Our automatic grader has a mean absolute error of 0.98 when compared to a human grader on a 9-point grading scale. This agreement is comparable to the average agreement between two human graders.