基于Canopy+K-means的中文文本聚类算法

摘要

随着互联网的发展,网络电子文本的数量急剧增加,给人们快速高效地从海量数据中挖掘出所需要的信息带来了巨大挑战.文本聚类是解决这个问题的一种可行方法.文章在文本聚类的过程中,针对K-means算法在聚类时需要事先指定簇的个数k和k个初始中心点这两方面的不足,采用Canopy+K-means的聚类算法进行中文文本聚类.为了提高K-means的聚类效果,先使用Canopy算法对数据进行"粗"聚类,在得到k值和聚类中心后,再使用K-means算法进行"细"聚类.在聚类过程中,为了避免"维灾难"现象,本文基于Word2vec通过获得同义词或近义词来有效减少文本特征向量的维度.实验结果表明,基于Canopy+K-means的聚类效果比传统的K-means算法有较好的纯度、准确率、召回率和F值.%With the development of the Internet,the amount of electronic texts has increased dramatically,which brings a huge challenge for people to dig out required information from the massive data.Text clustering is a feasible method to solve this problem,and K-means is a common algorithm for text clustering,which requires the researcher to specify the number of clusters k first and has sensitivity to the initial cluster centers. In order to improve the text clustering efficiency,Canopy+K-means algorithm is adopted.In detail,first Canopy is used for a"coarse"clustering,and then K-means algorithm is used for a"fine"clustering.In addition,Word2vec is used to obtain synonyms so as to reduce effectively the dimension of text feature vectors.It is found out that compared with K-means,Canopy+K-means obtains higher purity,precision,recall and F values.