Energy efficiency optimization in big data processing platform by improving resources utilization

摘要

Big data creates tremendous value for humanity, but also places a heavy burden on the environment. Energy consumption of computing platforms, especially big data processing platforms, cannot be ignored, and optimization of energy usage is imperative. To the out-of-core data processing, this paper proposes that the energy efficiency of a big data processing platform is closely related to the utilization of its computing resources; and an efficient resource allocation strategy for data processing tasks improves the platform's resources utilization. To improve the resources utilization, different resources are allocated according to a task-related Best Resource Ratio (BRR for short), such as "cpu, disk, network in the ratio of 1:2:4", rather than the resource's quantity, such as "cpu = 1 GHz, network = 20MB/s". We deduce the BRR of data processing tasks, and design a resource ratio based approach (R-2), which includes a task scheduling algorithm and resource allocation algorithm, for energy efficiency optimization. Experiments show that the R-2 approach can improve energy efficiency by 10%, 10% and 6% compared to FIFO, Capacity and Fair schedulers respectively, by maximizing resource utilization of a 12-node MapReduce platform. (C) 2018 Elsevier Inc. All rights reserved.