异构环境中HDFS数据块调度算法的设计与实现

摘要

Aiming at the insufficient of write performance of Hadoop Distributed File System(HDFS),a scheduling algorithm for concurrent transmission of HDFS data blocks in a heterogeneous environment is proposed.The algorithm monitors the resource status and memory queue of each node in the HDFS cluster in real time,matches receiving nodes with the forwarding nodes dynamically,makes the network cards and disk of the whole system work concurrently and reduces the time to write all copies to the distributed file system.The algorithm ensures that the data are written to disk before requesting the next data block for the data security.In the meantime,it makes the number of copies of each node match its own performance,so that the heterogeneous systems can achieve a high rate of writing.Performance tests show that the write performance of the distributed file system using the proposed algorithm is improved by 1 times compared with the original HDFS.%针对Hadoop分布式文件系统(HDFS)的写性能在执行效率上的不足,提出一种在节点性能异构环境中对HDFS数据块进行并发传输的调度算法.该算法实时监控HDFS集群中每个节点的资源状态和内存缓存队列,动态地将接收节点与转发节点进行配对传输,使全系统节点的网卡和磁盘并发工作,缩短了所有副本写入分布式文件系统的时间.将数据写入磁盘后请求下一个数据块,保证数据安全性,同时也使得各个节点获得与自身性能相匹配的副本数,使性能异构的系统能达到较高的写入速度.性能测试结果表明,使用该算法的分布式文件系统的写入性能较原始的HDFS提高了1倍.