Chunking-based Chinese Word Tokenization

摘要

This paper introduces a Chinese wordrntokenization system through HMM-basedrnchunking. Experiments show that such arnsystem can well deal with the unknown wordrnproblem in Chinese word tokenization.