Similarity-Based Node Distance Exploring and Locality-Aware Shuffle Optimization for Hadoop MapReduce

摘要

To shorten the networking delay from MapTracker to ReduceTracker has attractive potential to gain high performance shuffle for MapReduce. As the original MapReduce shuffle has no locality-aware feature when assigning reduce-tasks over computing nodes, we plan to present a similarity-based distance in the proposed Cloud Node Space to evaluate distance between two computing nodes in data center. Then, we implement a centralized and statistic-based locating service prior networking shuffle to place reduce-tasks near their corresponding data. Experimental results show that, comparing with Hadoop version, this service can achieve 2.3X speedup on shuffle time and bandwidth budget decreases by 60%.