A comparison study of clustering algorithms for microblog posts

摘要

Clustering is a popular unsupervised learning approach for topic analysis in text mining. In this paper, we do a comparison study of clustering algorithms for microblog posts, including weighting and programming model. Our experimental data is crawled from Sina Weibo in China. They are the 74,662 microblogs of 14 topics about Internet Technology. First of all, we do preprocessing to these microblog posts. Then we propose a manual sampling based dynamic incremental clustering algorithm (MS-DICA) to extract the topic threads from the microblogs we crawled. We evaluate the proposed algorithm from four aspects. Moreover, experimental comparisons are done in terms of accuracy and efficiency with the traditional k-means algorithm. Our experimental results show that the proposed MS-DICA is effective in the topic thread extraction. Its accuracy is close to the traditional k-means algorithm, and the running speed improves more than five times. In addition, the MapReduce programming model in Hadoop distributed computation platform that can run paralleled the k-means algorithm for cluster speeding up.