将维吾尔文从阿拉伯文、哈萨克文、柯尔克孜文等以阿拉伯字母为基础书写的类似文字中识别出来,是维文信息处理的基础.作者对维吾尔字符的编码优化后使用N元语法模型实现了维吾尔文的快速语种判别,准确率超过98％.经过错误分析,发现错误判别的文本主要集中在论坛和微博客中,这些文本有效字符数太少,语言特征不充分.最后作者计算了四种语言真实网络文本中的所有公共子串,并对文种判别所需要的最短字符串长度进行了分析.%Distinguishing Uyghur language from similar Arabic script languages such as Arabic, Kazakh, Kirgiz, etc. is an indispensable issue in Uyghur information processing. The paper builts a n-gram based Uyghur language discrimination model over an optimized Uyghur character encoding schema for an accuracy over 98%. The analysis reveals the misestimated texts are centered around the forum posts and microblogs because of their extremely short length (often only a few words). Thus, the paper examines all common sub-strings among tokens appeared in web texts of the four languages and probes into the minimum string length required to determine its language.
展开▼
