The Load Balance Strategy of MapReduce Jobs Based on the Fair Load Online Algorithm

摘要

Using MapReduce technology for parallel processing of data can improve the utilization of hardware devices and enhance the efficiency of data processing. However, MapReduce Technology is blind to the number of tasks in the frame, which will cause load imbalance of the node. Using the fair load online model to measure the performance of tasks slots can address the issue that the running speed of task nodes is not uniform due to the different CPU and hard disk speed of each node, as well as unknown input data and unsynchronized data between the MapReduce stages. It can distribute data according to the performance of the node so that each node load is more balanced, and it can be applied into the actual large-scale cluster. In this paper, the improved algorithm is placed in the load balancing and data evenly distributed jobs to verify its performance. Results show that the improved algorithm effectively improves the balance of the node load.