一种基于SA_LDA模型的文本相似度计算方法

摘要

Many information processing techniques are based on computing the similarity of text.However,the tradi-tional method of similarity calculation based on vector space model has the problems of high dimension and poor seman-tic sensitivity,so the performance is not very satisfactory.This paper proposed a self-adaptive LDA (SA_LDA)model based on traditional LDA model.It can manually determine the number of topic.Applying it in text similarity calcula-tion,it can solve the high dimensional and sparse problem.Experiments show that this method improves the accuracy of similarity calculation and the effect of text clustering compared with VSM.%计算文本的相似度是许多文本信息处理技术的基础.然而,常用的基于向量空间模型(VSM)的相似度计算方法存在着高维稀疏和语义敏感度较差等问题,因此相似度计算的效果并不理想.在传统的 LDA(Latent Dirichlet Allocation)模型的基础上,针对其需要人工确定主题数目的问题,提出了一种能通过模型自身迭代确定主题个数的自适应 LDA(SA_LDA)模型.然后,将其引入文本的相似度计算中,在一定程度上解决了高维稀疏等问题.通过实验表明,该方法能自动确定模型主题的个数,并且利用该模型计算文本相似度时取得了比 VSM模型更高的准确度.