SLA-Aware Energy-Efficient Scheduling Scheme for Hadoop YARN

摘要

Apache Hadoop becomes ubiquitous for cloud computing which provides resources as services for multi-tenant applications. YARN (a.k.a. MapReduce 2.0) is one of the key features in the second-generation Hadoop, which provides resource management and scheduling for large scale MapReduce environments. Two enormous challenges in the YARN scheduler are the abilities to automatically tailor and control resource allocations to different jobs for achieving their Service Level Agreements (SLAs), and minimize energy consumption of the overall cloud computing system. In this work, we propose an SLA-aware energy-efficient scheduling scheme which allocates appropriate amount of resources to MapReduce applications with YARN architecture. We perform job profiling to obtain the performance characteristics for different phases of a MapReduce application, which will be considered during resource provisioning in order to meet the completion deadlines specified by the application's SLA. Furthermore, an online userspace governor based dynamic voltage and frequency scaling (DVFS) scheme is designed in the YARN per-application ApplicationMaster to dynamically change the CPU frequency for upcoming tasks given the slack time between the actual execution time of completed tasks and expected completion time of the application. Experimental evaluation shows that our proposed scheme is both resource and energy efficient compared with the existing MapReduce scheduling policies.