Big Data Equi-Join Optimization Algorithms on Spark Cloud Computing Platform

摘要

On Spark cloud computing platform, the conventional big data equi-join algorithms cannot meet the performance requirements well and the procedure of it is very time-consuming, so the efficiency of big data equi-join is a burning challenge. To overcome it, in this paper, we propose Compressed Bloom Filter Join algorithm, an efficient algorithm filters out most of invalid connections which cannot meet the criteria to reduce network overhead, and it constructs static one-dimensional bit array to improve join performance. Moreover, Compressed Bloom Filter Join Extension algorithm, an extended optimization based on Compressed Bloom Filter Join algorithm, produces a dynamic two-dimensional bit array to filter out invalid records, and it can further accelerate the process of data join when the data size is unknown. Experimental results show that the performance of two optimization algorithms which can reduce time consumption and the data size of Shuffle stage are better than Hash Join and Broadcast Join on Spark cloud computing platform.