Models and Algorithm of Chinese Word Segmentation

摘要

Chinese word segmentation is of significance in Chinese Natural Language Processing (NLP). This paper proposes a statistical segmentation model, which integrates the character juncture model (CJM) with word bi-gram language model, and then designs a strategy for making an accurate and low-cost estimation for this model. The advantage of the proposed model is that it can employ the affinity of characters inside or outside a word and word co-occurrence information simultaneously to handle ambiguity. After investigating the differences between real and theoretical size of segmentation space, we apply A~* algorithm to perform segmentation without exhaustively searching all the potential segmentations. Experiments show that the proposed methods are efficient, achieving over 92% correct disambiguation and 84% unknown word correct identification respectively in our preliminary tests.