同构Hadoop环境作业执行时间计算方法

摘要

Execution time is very important for job scheduling. In this paper, the execution characters of Hadoop MapReduce jobs are analyzed, and then a new method is proposed to compute the execution times of these jobs. The method takes the execution times of map task and reduce task as input data. It captures these execution times by pre-executing under an assumption. The method has been evaluated in a Linux cluster, the experiment results show that the method computed the execution times of jobs with the error rate no more than 7%.%执行时间是作业调度的重要参考因素之一。通过分析Hadoop MapReduce环境作业的执行特征，提出了以map任务和reduce任务执行时间为输入，估算作业执行时间的方法。该方法在一定假设条件下，借助作业预执行来获取map任务和reduce任务的执行时间。实验结果表明，该方法估算作业执行时间的误差率小于7%。