Improved YARN resource scheduling algorithm based on network load sensing

摘要

Confronted with multitask scheduling, YARN may conduct a precise calculation and allocation towards every single task's computing and memory resources, however it failed to consider the impact on task fulfillment and cluster resources utilization. In this paper, we propose a new algorithm to allocate the data interconnected tasks to the same node or to another closely-located low network load node. This new algorithm firstly builds a task flow graph for the task requests of an application, and then assigns tasks more properly using the greedy strategy, based on the state of network resource. In this way, it significantly improves the management performance of YARN in the case of network congestion or instability. Besides, it consumesless execution time, makes higher utilization of CPU and requires lower network load.