A Density-Aware Similarity Join Query Processing Algorithm on MapReduce

摘要

Recently, the amount of data is rapidly increasing and thus MapReduce has attracted much interest as a new paradigm for such data-intensive applications. Similarity join is an essential operation for data analytics, including record linkage, near duplicate detection, document clustering. However, the performance of MapReduce is limited when applied on complex data analytical task involving joins of multiple datasets. Hence, workload-aware data partitioning techniques are required, which ensure the balance of computation of each machine. In this paper, we propose a similarity join algorithm using MapReduce that provides scalability and high performance by using grid-based data mapping technique for joining datasets. From the experiment analysis, we prove that our algorithm outperforms the existing algorithm under various data size and similarity thresholds.