大规模短文本的不完全聚类

摘要

聚类分析是数据挖掘的一个重要手段,人们可以通过聚类发现信息中潜在的热点或规律.至今,已经有大量聚类算法被研究和提出.随着互联网的日益普及,查询日志、Twitter等短文本信息逐渐在人们生活中起着越来越重要的作用.这类短文本信息数量巨大,通常可达到千万乃至亿级,现有的聚类算法在对这类大规模短文本信息进行聚类分析时往往显得异常无力.该文通过对实际应用中的短文本信息进行实验分析,发现了这类数据类别所具有的"长尾现象",并由此提出了不完全聚类思想,可以有效地提高这类短文本信息的聚类性能.%Clustering is an unsupervised classification of patterns (observations, data items, or feature vectors) into groups (clusters).So far, many clustering algorithms have been proposed.With the rapid development of internet,short texts such as query logs and Twitter messages play a more and more important role in our daily life.Most existing clustering methods are hard to be applied in dealing with this kind of information due to the huge scale of data.This paper reveals the long tail distribution of this kind of information, and proposes an incomplete clustering algorithm.The experimental results show that the proposed method can cluster the short texts effectively and efficiently.