Design and Implementation of a Two Level Scheduler For Hadoop Data Grids

摘要

Hadoop is a large scale distributed processing infrastructure designed to handle data intensive applications. In acommercial large scale cluster framework, a scheduler distributes user jobs evenly among the cluster resources. Theproposed work enhances Hadoop’s fair scheduler that queues the jobs for execution in a fine grained manner using taskscheduling. In contrast, the proposed approach allows backfilling of jobs submitted to the scheduler. Thus job level and tasklevel scheduling is enabled by this approach. The jobs are fairly scheduled with fairness among users, pools and priority.The outcome of the proposed work is that short narrow jobs will be executed in the slot if sufficient resource is not availablefor larger jobs. Thus shorter jobs get executed faster by the scheduler when compared to the existing fair scheduling policythat schedules tasks based on their fairness of remaining execution time. This approach prevents the starvation of smallerjobs if sufficient resources are available.