基于 Hadoop MapReduce并行近似谱聚类算法研究与实现

摘要

With the advent of information age, the large-scale high-dimensional data generated in Internet increases exponentially, its spectral clustering suffers from the bottleneck problem in both computational time and memory use, particularly in solving Laplacian matrix eigenvector decomposition.Given the advantages of Hadoop MapReduce parallel programming model in processing intensive data, based on t nearest neighbour sparse approximation similarity Laplacian matrix, in this paper we design Hadoop MapReduce parallel approximate spectral clustering algorithm to solve the above-mentioned bottleneck problem.The experiment uses UCI Bag of Words dataset to validate the correctness and effectiveness of the designed algorithm, result indicates that the parallel design aligns with a certain desired effect in terms of spectral clustering quality and performance.%随着信息时代的来临，互联网产生的大规模高维数据呈现几何级数增长，对其进行谱聚类在计算时间和内存使用上都存在瓶颈问题，尤其是求Laplacian矩阵特征向量分解。鉴于Hadoop MapReduce并行编程模型对密集型数据处理的优势，基于t最近邻稀疏化近似相似Laplacian矩阵，设计Hadoop MapReduce并行近似谱聚类算法，以期解决上述瓶颈问题。实验使用UCI Bag of Words数据集验证所设计算法的正确性和有效性，结果显示该并行设计在谱聚类质量和性能方面达到了一定的预期效果。