A Chinese Word Segmentation Based on Machine Learning

摘要

Different from English, there are no interval marks between words in Chinese. Segmenting Chinese text to words is the first job for every kind of Chinese information processing, so Chinese word segmentation is a basal and difficult issue in the field of Chinese information processing. Traditional word segmentation systems have to establish the dictionary and add unknown words out of the dictionary with manual work. This paper proposes a new Chinese word segmentation model which can automatically establish a dictionary, gradually update it and perfect it based on machine learning. Four modules of the machine learning model for Chinese word segmentation system are introduced in detail and some improvements of the algorithms are made on some module to improve system performance. After the test of closed corpus and open corpus, the results show that the method alleviates the workload of building and maintaining the dictionary, furthermore, it resolves the issues of ambiguity processing and unknown words recognition.