Proposing a Method for Evaluating Text Summarization Systems

摘要

Nowadays summarizers and machine translators have attracted much attention to themselves, and many activities on making such tools have been done around the world. For Farsi like the other languages there have been efforts in this field. So evaluating such tools has a great importance. Human evaluations of machine summarization are extensive but expensive. Human evaluations can take months to finish and involve human labor that cannot be reused. In this paper, we propose a method of automatic machine summarization evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. This method has the metrics of determining auto summaries' quality, through comparing them to the summaries produced by Human (ideal summaries). These metrics measures overlapping of system summaries and human ones in number of units like n-tuples, words string and pairs of words. Certainly for semantic comparing of texts in case of review summaries, the appearance of words are not enough and using of WordNet seems to be necessary. In the proposed method words network is used with an appropriate idea and has improved evaluation results significantly. The proposed method is the first method for the Persian language. Performance measurement of the tool was done during a specified and standard procedure and the results indicate acceptable yield of it. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.