A distributed parallel algorithm for inferring hierarchical groups from large-scale text corpuses

摘要

We propose a distributed parallel algorithm for inferring the hierarchical groups present in arnlarge-scale text corpus. The algorithm is designed to deal with corpuses that typically do not fitrninto the main memory of a workstation computer. The key contribution of this paper lies in itsrnproposal and verification of a parallel distributed algorithm that exploits the advantages of tworncomplementary techniques based on (i) localized modularity optimization and (ii) spectral clustering.rnBased on our experimental observations, these are complementary in the sense that thernformer excels at finding coarse groups in a large-scale network, while the latter demands a heavyrnmemory footprint but is effective in inferring tightly knit fine-grained groups. Empirical evaluationrnof the distributed implementation scheme shows that the algorithm exhibits a significantrnspeed-upwhen compared to existing algorithms like Louvain and, at the same time, produces betterrnquality clusters than either Louvain or spectral clustering algorithms in terms of the F-scorernand Rand index.