Parallelizing K-Means-Based Clustering on Spark

摘要

K-means is in fact a family of clustering algorithms with different distance functions and a variety of extension, e.g., fuzzy clustering and consensus clustering. Nevertheless, K-means-based clustering algorithms employ the similar two-phase iterative procedure including distance computation and centroids updating. Therefore, to explore the parallel implementations of this two-phase iterative procedure on Spark is not only universal to a wealth of clustering algorithms but also meets the practical needs addressed by big data. This paper contributes to reveal implementation details for parallelizing K-means-based clustering on Spark. In particular, we first introduce the boundary of so-called K-means-based clustering, and then present the overall parallelizable framework on Spark. We discuss the technical barrier and their alternative strategies for each step. Experimental results on both large-scale UCI datasets and text datasets demonstrate the effectiveness and efficiency of our implementations.