Integrating Probability into LR Parsing

摘要

In this paper, we describe the methods of acquiring stochastic knowledge from corpus and integrating both of the lexical and syntactic probabilities into a LR parser in order to improve its disambiguatign ability. Based on the Hidden Markov Model (HMM) tagging system, we introduce the lexical statistic information acquistion. These information can be learned from the corpus no matter the corpus is tagged or not. On the syntactic(grammar) level, we proposed two different concepts; derivation porbability and reduction probability of Contect Free Grammar(CFG) rule.Because the original Inside-OUtside algorithm can only estimate the derivation probabilities of rules, we designed and implemented an automatic method to estimate the reduction porbabilities of rules from corpus. We also applied both the lexical porbability and rule probability of grammar to the CFG parser of a English to Chinese MT system. We set up a new kind of scoring system based on the statistic knowledge as the criteria of disambiguating the syntactic structures. Experiment shows that the accuracy of the parsing has been improved.