A Japanese Morphological Analysis Method Using a Statistical Language Model and an N-best Search Algorithm

摘要

We present a novel method for Japanese morphological analysis which uses a statistical language model and an N-best search algorithm. It has a probabilistic model for unknown words to parse unrestricted Japanese sentences accurately and it can get N-best morphological analysis hypotheses. When the statistical Japanese morphological analyzer was trained on the subset of the EDR corpus (about 190 thousand sentences, 4.7 million words) and tested on 100 sentences of open text, it achieved 94.6/100 recall and 93.5/100 precision for the top candidate, and 97.8/100 recall and 88.3/100 precision for the top five candidates.