Implementation of a Hadoop platform scheduling algorithm based on a genetic algorithm

摘要

A job scheduling algorithm is one of the most important key technologies for the Hadoop platform, directly affecting the overall performance and the utilization rate of a system resource. With the traditional Hadoop three scheduling algorithms in the face of a large number of short jobs, the processing speed is not satisfactory. In this paper, the improvement of existing scheduling algorithms, considering the job point of view, the task decomposition of short work some of the first as a queue, and based on the genetic algorithm provides the solution space to select the operation sequence, to avoid the frequent switching requires time overhead of shorter jobs between; at the same time, adapting long jobs and setting up a task queue length limit to meet demand. The experiment results demonstrate that the proposed algorithm is an efficient job scheduling algorithm.