基于延迟调度策略的reduce调度优化算法

摘要

在大规模的Hadoop集群中,良好的任务调度策略对提高数据本地性、减小网络传输开销、减少作业执行时间以及提高集群的作业吞吐量都有着重要的影响.针对Hadoop架构中reduce任务的数据本地性较低问题,提出了一种基于延迟调度策略的reduce任务调度优化算法,通过提高reduce任务的数据本地性来减少作业执行时间以及提高作业吞吐量,该算法在Hadoop架构的early shuffle阶段,使用多级延迟调度策略来提高reduce任务的数据本地性.最后重写原生公平调度器代码实现了该调度算法,并与原生公平调度器进行了对比实验分析.实验结果表明,该算法明显减少了作业执行时间,提高了集群的作业吞吐量.%In large scale Hadoop cluster,good task scheduling strategy is important to improve data locality,reduce network transmission overhead,reduce job execution time and improve job throughput.In view of the low data locality problem of reduce task in Hadoop architecture,this paper put forward a reduce task scheduling optimization algorithm based on delay scheduling policy,which reduced the job execution time and improved the job throughput by improving the data locality of the reduce task.In the shuffle early phase,the algorithm used a multi-stage delay scheduling policy to improve the data locality of the reduce task.This paper rewrote the native fair scheduler code to realize the scheduling algorithm,and conducted contrast experiment with native fair scheduler.Experimental results show that the proposed algorithm significantly reduces the job execution time,and improves the job throughput.