基于词频学习和动态词频更新的藏文自动分词系统设计

摘要

Automatic Tibetan word segmentation is one of the basic problems in natural language processing of Tibetan.In this paper,we design a new automatic Tibetan word segmentation system in light of the keys and difficulties in it,for example:the technologies of identification of case-auxiliary word,the ambiguity segmentation,and the unknown words recognition.The system uses the techniques of the dynamic word frequency up-date and the ambiguity treatment and unknown words recognition which are based on the word frequency of the context.The presented system has relatively high performance in terms of the recognition accuracy of ambiguities,the recognition rate of unknown word and the segmentation speed.%藏文自动分词问题是藏文自然语言处理的基本问题之一。针对藏文自动分词中的重点难点，例如：格助词的识别、歧义切分、未登录词识别技术设计一个新的藏文自动分词系统。该系统采用动态词频更新和基于上下文词频的歧义处理和未登录词识别技术。在歧义字段分词准确性、未登录词识别率和分词速度上，该系统具有较优的性能。