Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure

摘要

Recently, MapReduce has been a popular distributed programming framework, which divides a job into map tasks and reduce tasks and executes these tasks in parallel over a large-scale MapReduce cluster to speed up job execution. Generally, the cluster is a master-slave infrastructure. To prevent jobs from being interrupted due to node failure, current MapReduce implementations, such as Hadoop, adopt a task-reexecution policy on the slave side, i.e., when a slave node due to failure cannot complete a task, this task will be reassigned to another available slave for reexecution. However, on the master side by default, no redundancy scheme is provided. Since this type of infrastructure has been worldwide adopted, we call it the general MapReduce infrastructure (GMI). To achieve a more reliable and energy-efficient working environment, understanding the impact of GMI on its job completion reliability (JCR) and job energy consumption (JEC) is required. In this paper, we base on a Poisson distribution to analyze GMI's JCR from a single-job perspective. After that, we accordingly derive the corresponding JEC. Through the analytical results, MapReduce managers can comprehend how GMI behaves and how their MapReduce can be improved so as to achieve a more reliable and energy-efficient MapReduce environment.