基于MapReduce框架的重分区连接的优化研究

摘要

Repartition join is the most commonly used join strategy on MapReduce framework. This paper discusses the standard repartition join and the improved repartition join which fixed the buffering problem of the standard version. And then, a new method, pre-filter is proposed. The pre-filter runs at the stage when MapReduce divides the large date file into several splits. The method can make the amount of data smaller in every stage of MapReduce processing, save the space of buffer, and reduce the overhead of data transmission between Map stage and Reduce stage.%重分区连接查询是基于传统MapReduce框架的最常用的连接查询算法之一。在讨论基于传统MapReduce框架的标准重分区连接算法及减小数据缓存的改进算法的基础上，提出了在数据文件分块阶段进行预筛选以精简MapReduce框架中处理的数据量的方法。该方法能有效减少框架内部各个阶段处理的数据总量，进一步压缩缓存的使用空间并降低不同阶段之间数据传输的网络开销。