H adoop延迟调度中延迟时间间隔的合理设置

摘要

In Hadoop framework-based cloud computing , in order to reduce data transfer and improve program efficiency , the delay scheduling algorithm allows job to spend a certain amount of delay time interval to wait the data to be processed in the operation to be contained in certain computing resource , but the delaying time interval selected is often an experience value .In this paper we introduce the probability of parameter users expected localisation and derive the calculation formula of waiting time based on the analysis of how the operating data to be processed in the file system affects the localised job scheduling .The formula distinguishes different jobs and sets different waiting time, and the users can control the localisation degree the job expected based on the parameter of desired localisation probability . Experiment is conducted to verify the above method , the results show that the delay time computed by the above formula can make the operation reach the localisation effect that the users expected .%基于Hadoop框架的云计算中，为减少数据迁移提高程序执行效率，延迟调度算法允许作业花费一定的延迟时间间隔等待某计算资源包含该作业待处理数据，而延迟时间间隔的选择往往是一个经验值。在分析了作业待处理数据在文件系统中的分布情况如何影响作业本地化调度的基础上，引入参数用户期望本地化概率，推导出等待时间的计算公式。该公式区分不同的作业，设置不同的等待时间，并且用户可以根据期望本地化概率这一参数来调控作业预期的本地化程度。对上述方法进行实验验证，结果表明：通过公式计算出的延迟时间能够使得作业达到用户预期的本地化效果。