Segmental HMM-based Part-of-speech Tagger

摘要

This paper presents a solution in order to solve the problem of using HMM-based POS tagger in some languages where a word can be comprised of several tokens. Viterbi algorithm is modified in order to support segment of words within a model state. In the other word, the proposed system has a built-in tokenizer where indicates words boundaries as well as its corresponding tag sequence.