基于主题和特征的文本相似度算法研究

摘要

In this paper, we proposes a texts’ similarity algorithm based on the topic and features of each text, the purpose is to accurately mine the nearest neighbor texts of a given text. Firstly, we introduces the characteristics of CHI Square Statistics and gives improvement to features selection of training text which have known topic; and then com-pare the minimum edit distance algorithm (LevenShtein Distance), Cosine Similarity Algorithm and Jaccard Similarity Coefficient, analyze principal defect of each algorithm and propose a new text similarity algorithm based on features and topic; after that we use the new algorithm on real data set and prove no matter in speed or accuracy the new one is better than others; In the end, the new algorithm is applied to stocks’ subject tagging, from the analysis of results, we expound recent shortage and put forward the improvement.%本文提出了结合主题和各主题下关键特征的文本相似度算法，目的在于更准确的挖掘被描述对象的近邻对象集。本文首先介绍卡方统检验特征统计法，并利用改进的卡方检验，计算训练集中已知主题的文本的特征；而后介绍了最小编辑距离算法、余弦相似度算法和杰卡德相似系数，在论证了主题对文本相似度的重要性后，又针对难提取主题的文本加以改进，最终提出了基于主题和特征的文本相似度算法；然后对各个算法在测试集上的相似度计算结果进行分析，证明本文提出的算法在速度和精确度上明显优于其他算法；最后将该算法应用于股票的概念股题材标注上，分析结果并提出改进空间和不足之处。