Ancient Chinese Lexicon Construction Based on Unsupervised Algorithm of Minimum Entropy and CBDB Optimization

摘要

Ancient Chinese text segmentation is the basic work of the intelli-gentization of ancient books. In this paper, an unsupervised lexicon construction algorithm based on the minimum entropy model is applied to a large-scale ancient text corpus, and a dictionary composed of high-frequency co-occurring neighbor characters is extracted. Two experiments were performed on this lexicon. Firstly, the experimental results of ancient text segmentation are compared before and after the lexicon is imported into the word segmentation tool. Secondly, the words such as person's name, place name, official name and person relationship in CDBD are added to the lexicon, and then the experimental results of ancient text segmentation before and after the optimized lexicon is imported into the word segmentation tool are compared. The above two experimental results show that the lexicon has different enhancement effects on the segmentation effect of ancient texts in different periods, and the optimization effect of CDBD data is not obvious. This article is one of the few works that applies monolingual word segmentation to ancient Chinese word segmentation. The work of this paper enriches the research in related fields.