Improving Chinese Word Segmentation with Description Length Gain

摘要

Supervised and unsupervised learning has seldom joined with and thus lend strength to each other in the field of Chinese word segmentation (CWS). This paper presents a novel approach to CWS that utilizes description length gain (DLG), an empirical goodness measure for unsupervised word discovery, to enhance the segmentation performance of conditional random field (CRF) learning. Specifically, we attempt to integrate the lexical information acquired from the unsupervised DLG segmentation into the supervised CRF learning of character tagging for CWS. Our experimental results show that the CRF learning can be further improved on top of its state-of-the-art performance in the field by making good use of DLG information.