A Multi-granulariry Chinese Word Segmentation Method Based on Semantic Similarity for Risk Sources

摘要

This paper presents a multi-granularity Chinese word segmentation method based on semantic similarity for risk sources. Based on expert knowledge, a risk source root dictionary and a risk source extension dictionary were established, and the risk source extension dictionary was trained using the LDA model. Firstly, the sentence containing the root vocabulary was divided into two sentences using root dictionary. Two semantic segment Word next to the root vocabulary were combined into two new extension vocabulary, which were calculated semantic similarity by Latent Dirichlet Allocation model together with the root vocabulary. Finally, the word with the largest similarity was used as the extension risk word. Experiment show that the average success rate of our method reached 94.19%, and can be used for text segmentation of criminal cases.