Research of segmentation of Chinese texts in Chinese search engine

摘要

Segmenting Chinese texts into Chinese words is a very difficult problem. In this paper, a framework for a Chinese Internet search engine is presented. It discusses the characteristics and difficulties of segmentation of Chinese texts in Chinese search engines. The paper concludes that the correctness of Chinese segmentation is most important, and puts forward tactics for processing disambiguation of segmentation strings, new unknown words and stop words, and presents methods which satisfy the consistency of Chinese segmentation.