Using Hidden Markov Model to improve the accuracy of Punjabi POS tagger

摘要

POS tagger is the process of assigning a correct tag to each word of the sentence. Accuracy of all NLP tasks like grammar checker, phrase chunker, machine translation etc. depends upon the accuracy of the POS tagger. We attempted to improve the accuracy of existing Punjabi POS tagger. This POS tagger lacks in resolving the ambiguity of compound and complex sentences. A Bi-gram Hidden Markov Model has been used to solve the part of speech tagging problem. An annotated corpus of 20,000 words was used for training and estimating of HMM parameter. Maximum likelihood method has been used to estimate the parameter. This HMM approach has been implemented by using Viterby algorithm. A module has been developed that takes the existing POS tagger output as input and assign the correct tag to the words having more than one tag. Our module was tested on the corpus containing 26,479 words. The accuracy of 90.11% was evaluated using manual approach.