Evaluation of text summaries without human references based on the linear optimization of content metrics using a genetic algorithm

摘要

The Evaluation of Text Summaries (ETS) has been a task of constant challenges to the development of Automatic Text Summarization (ATS). Within the ATS task, the ETS is crucial to determine the performance of text summaries. Over the last two decades, the scientific community has used the ROUGE system as a standard package to assess the content of automatic summaries. However, if there are not human-made summaries (called human references), then the evaluation cannot be carried out. For this reason, the different state-of-the-art evaluation methods have been proposed that analyze the summary content using the source documents. Nonetheless, these methods do not highly correlate with human assessment. In this paper, a linear optimization of content-based metrics is proposed using a Genetic Algorithm (GA) to improve the correlation between automatic and manual evaluation. The proposed method combines 31 content metrics based on the evaluation without human references. The results of the linear optimization show correlation improvements concerning other evaluation metrics on DUC01 and DUC02 datasets.