Evaluation of automatic summarization methods by being compared with manually summarized texts on BBS survey web sites

摘要

Many people communicate by using Internet. The amount of log of their communication is growing very rapidly. However we have a potential desire to know what people say, undoubtedly it is difficult to read all of the log. Therefore, people access portal sites, web survey sites, and so on. In Japan there are several popular mega Bulletin Board System (BBS) and social media that are too wide to be looked through. Therefore some text summarization researches try to summary the communication log automatically. Text summarization is one of popular research themes in AI but there is no complete method. One of the difficulties is that there is no good evaluation methods. Actually, the evaluation of the past researches is not enough. There are a few mathematical methods. Questionnaire is unstable and costly but this is still one main evaluation method. Usually it is difficult to collect a sufficient number of test subjects for the evaluation of text summarization techniques. In this paper, we propose a new evaluation method for text summarization of log of communication. Recently, many web sites which manually survey and summary their communication log are managed. These web sites pick up interesting conversation on log of BBS and modified it more attractively. We think that summaries on these web sites are very useful and more reliable than traditional questionnaire because a lot of pair of an original log and its handmade summary can be obtained. In this paper, we show the result of F-measure between summaries generated by common text summarization techniques and the BBS survey sites' summaries. As a result, we can say that undiscovered human criteria for summarization should be found.