Recognition of Out-of-vocabulary Words Using Stopwords

摘要

Main impact in Chinese word segmentation is from disambiguation and recognition of out-of-vocabulary (OOV) word. Studies show that the accuracy in word segmentation caused by OOV words is reduced more seriously than that by ambiguous words. Therefore, Chinese word segmentation would be improved largely and effectively by the correct recognition of OOV words. Though the recognition of OOV words is impacted by stopwords, it is conductive to improve the quality and the efficiency in the segmentation process with a proper use. Motivated by this, a recognition algorithm of OOV word based on stopwords, named ROWS, is proposed in this paper. It achieves the prominent improvement of the recognition of OOV words. Extensive studies demonstrate that in comparison with the state-of-the-art segmentation method of ICTCLAS, ROWS outperforms on the recognition of OOV words, and the values of Recall and Precision are improved by 5.65and 4.8% respectively on average.