基于链式条件随机场的中文分词改进方法

摘要

With the development of Chinese word segmentation evaluation Bakeoff,the word-position-based tagging Chinese word segmentation approaches based on chain conditional random fields have been widely used.For the training of CRF models,it is essential to select the tone tag set and feature template.However,the researches in the literature generally used single tag set or feature template,lacking of frequently-used tag sets and feature templates in combination,which resulted in out-of-vocabulary rate at a low level and influenced the performance of word segmentation on Internet corpuses.This method firstly combines six-tag set and feature template TMPT-10 and TMPT-10`, dealing with comparative experiments with frequently-used tag sets and feature templates on the Bakeoff corpuses.The results demonstrate that the improved method 6tag-tmpt10 can reach higher rate of out-of-vocabulary word recall compared with other methods,which can improve the performance of Chinese word segmentation in Internet field,in the meanwhile can get the comparative F1-score.%基于链式条件随机场模型的序列标注中文分词方法随着中文分词评测 Bakeoff 的展开得到广泛应用。词位标注集和特征模板集对该模型的学习至关重要，但当前的研究大多采用单一的标注集和特征模板集进行实验，缺乏标注集和特征模板集结合的尝试，使得中文分词中未登录词识别率不高，从而影响互联网领域语料的分词效果。首次采用六词位标注集结合 TMPT-10和TMPT-10｀特征模板，并与常见标注集和特征模板集的组合在 Bakeoff 语料上进行实验对比，结果表明，改进的方法6tag-tmpt10取得更好的未登录词召回率，在互联网领域中文分词能取得很好的效果；同时在 F 值上也与其他最好结果相当。