New information theoretic measures for text extraction

摘要

In this paper we propose two information theoretic measures called collocation information (CI) and discrimination threshold (DT). These two measures are used to quantify the relationship between collocating words. CI of a word is derived from Shannon's measure for information while DT is the result of a constrained optimization in CI. Properties of CI and DT are studied and their application in generating sentence extracts as text document summaries is discussed. Extracts generated using these measures are evaluated using automatic evaluation schemes and their performance is compared with other known schemes.