词典与统计方法结合的中文分词模型研究及应用

摘要

为了解决传统的基于词典的分词法和基于统计的分词方法的效率和识别能力的不足,根据电子商务中商品名称信息这一特定领域的文本数据的特点进行分析,研究了mmseg分词法和基于互信息的处理方法,结合两类分词方法的优点,将mmseg分词算法和互信息的算法应用于分词处理过程中,设计并实现了一个快速、准确度高的分词模型,通过测试结果表明,该模型能够较好地解决分词的速度与效率问题.%To solve the problem that there is a lack of efficiency and recognition ability in the dictionary-based word segmentation method and in the statistical-based word segmentation method, the specific areas of product name text data in E-commerce is ana-lyzed, and the "mmseg" word segmentation method and mutual information processing method are researched, A rapid and highly accurate word segmentation model is designed and proposed, two types of word segmentation method are untilled, and "mmseg" segmentation algorithm and mutual information segmentation algorithm are applied in word segment processing. The test proves that this model can provide a better solution for segmentation speed and efficiency.