A Unicode based Adaptive Segmentor

摘要

This paper presents a Unicode basedrnChinese word segmentor. It can handlernChinese text in Simplified, Traditional, orrnmixed mode. The system uses the strategyrnof divide-and-conquer to handle thernrecognition of personal names, numbers,rntime and numerical values, etc in the preprocessingrnstage. The segmentor furtherrnuses tagging information to work onrndisambiguation. Adopting a modularrndesign approach, different functional partsrnare separately implemented usingrndifferent modules and each modulerntackles one problem at a time providingrnmore flexibility and extensibility. Resultsrnshow that with added pre-processingrnmodules and accessorial modules, thernaccuracy of the segmentor is increasedrnand the system is easily adaptive torndifferent applications.