Improving MapReduce energy efficiency for computation intensive workloads

摘要

MapReduce is a programming model for data intensive computing on large-scale distributed systems. With its wide acceptance and deployment, improving the energy efficiency of MapReduce will lead to significant energy savings for data centers and computational grids. In this paper, we study the performance and energy efficiency of the Hadoop implementation of MapReduce under the context of energy-proportional computing. We consider how MapReduce efficiency varies with two runtime configurations: resource allocation that changes the number of available concurrent workers, and DVFS (Dynamic Voltage and Frequency Scaling) that adjusts the processor frequency based on the workloads' computational needs. Our experimental results indicate significant energy savings can be achieved from judicious resource allocation and intelligent DVFS scheduling for computation intensive applications, though the level of improvements depends on both workload characteristic of the MapReduce application and the policy of resource and DVFS scheduling.