Filtering of Word Bigrams by Considering Similar Bigrams as Evidence of Unimportance

摘要

Compound terms play important roles in may applications. But there are many trivial terms which are mostly useless and sometimes harmful. In this paper, we focus on word bigrams rather than general n-grams, and describe a new method of distinguishing important word bigrams from trivial ones. This method characterizes trivial bigrams as those having remarkably similar ones. By automatically discarding trivial bigrams using this method, we can reduce considerably the amount of human work needed for selecting important bigrams. In an experiment in which important bigrams are selected from Japanese newspaper articles, we were able to reduce about 25% of human work with almost no loss of important bigrams.