一种基于交互式的Hadoop作业调度算法

摘要

Hadoop平台中作业调度是一个重要环节。 FIFO是Hadoop默认的调度算法,简单易实现且应用广泛,但其在数据的本地化( data locality)这一特性上考虑不足,会引起网络的负载量增大,任务的等待执行时间长,计算资源得不到充分利用等一系列弊端；同时Map阶段和Reduce阶段资源槽的静态职能形式也更一步加深了这种缺陷。针对这些缺陷,从数据的本地性、任务分配的角度出发,提出了一种基于主从节点间交互的作业调度算法( Interactive Scheduler,IS)。该算法是对FIFO的一种改进,同时也使不同资源槽之间可以动态转换,提高了资源的使用率。通过实验对比,结果表明IS调度算法对Hadoop平台的作业调度效率有显著的提升。%Job scheduling is an important part of Hadoop. FIFO,as a scheduling algorithm by Hadoop,is simple and easy to achieve and widely used,but it is lack of consideration in the characteristic of data locality,that will cause network transmission increased and task waiting long execution time and computing resources cannot be fully utilized and a series of drawbacks. Meanwhile the static function of resource slots in Map and Reduce stages further increases the defects. So a job scheduling algorithm ( Interactive Scheduler,IS) based on interacting the master node and slave nodes from the data locality and tasks allocation is proposed,which is improvement for FIFO,and realizes the dynamic conversion of map slots and reduce slots,and increases the usage of resources. Through the comparison of experi-ment,it proves that the IS has a great improvement in job scheduling for Hadoop.