A Japanese Word Segmenter Using a PPM-based Language Model

摘要

Word segmentation, which segments an input sentence into words, is the most fundamental process of Japanese language processing. In this paper, we present a new method for segmenting the input sentence into words, which is suitable for those languages that have no delimiter between words, such as Japanese and Chinese. First, we present a word segmentation model using a character--based n-gram model, which is our baseline method. Next, we apply the PPM compression algorithm to the problem of word segmentation. PPM (Prediction by Partial Matchiflg) is a lossless compression algorithm based on a finite-context probabilistic modeling technique and PPM is a variant of PPM, in which there is no a priori bound on con- text length. As the result of experiments on the ADD (ATR Dialogue Database) corpus, the proposed Japanese word segmenter using the PPM--based language model marked a higher accuracy than the character-based n-gram model. In particular, the proposed method using the PPM-based language model achieved 97.67/100 recall and 98.27/100 precision for open text.