基于Spark的大规模文本k-means并行聚类算法

摘要

互联网文本数据量的激增使得对其作聚类运算的处理时间显著加长,虽有研究者利用Hadoop架构进行了k-means并行化研究,但由于很难有效满足k-means需要频繁迭代的特点,因此执行效率仍然不能让人满意.该文研究提出了基于新一代并行计算系统Spark的k-means文本聚类并行化算法,利用RDD编程模型充分满足了k-means频繁迭代运算的需求.实验结果表明,针对同一聚类文本大数据集和同样的计算环境,基于Spark的k-means文本聚类并行算法在加速比 、扩展性等主要性能指标上明显优于基于Hadoop的实现,因此能更好地满足大规模文本数据挖掘算法的需求.%Due to sharp increase of internet texts ,the processing of k-means on such data is incredibly lengthened . Some classic parallel architectures ,such as Hadoop ,have not improved the execution efficiency of K-means ,because the frequent iteration in such algorithms is hard to be efficiently handled .This paper proposed a parallelization algo-rithm of k-means based on Spark .It makes full use of in-memory-computing RDD model of Spark so as to well meet the frequent iteration requirement of k-means .Experimental results show that k-means executes much more effi-ciently in Spark than in Hadoop on the same datasets and the same computing environments .