A Load-Balancing Algorithm for Hadoop Distributed File System

摘要

Hadoop Distributed File System (HDFS) is developed to store a huge volume of data. Files are divided into blocks and the replicated blocks are then stored on many DataNodes in a distributed manner. Although doing so makes HDFS fault tolerant, the random nature of the default block placement strategy may lead to load imbalance among the DataNodes. Moreover, the built-in load-balancing algorithm Balancer may reduce the performance and consume lots of network resources. Therefore in this paper we consider all the situations that may influence the load-balancing state and propose a new load-balancing algorithm. In the proposed algorithm a new role named BalanceNode is introduced to help in matching heavy-loaded and light-loaded DataNodes, so those light-loaded nodes can share part of the load from heavy-loaded ones. The simulation results show that our algorithm can achieve a good load-balancing state in the HDFS compared with two existing algorithms.