An locality-aware scheduling based on a novel scheduling model to improve system throughput of MapReduce cluster

摘要

Scheduling algorithms place a crucial role in MapReduce systems. Several recent scheduling algorithms, however, are all under Job-Task scheduling model which makes task scheduling confined, leading to poor task scheduling preference such as data locality, scan sharing and etc. These characteristics are very important heuristics on data intensive computing and helpful in improving system throughput. In this paper, we firstly design a novel scheduling model termed as Tasks-Job scheduling to overcome the above issues. Furthermore, we propose a locality aware algorithm to improve system throughput. Comprehensive experiments have been conducted to compare the proposed scheduling model and algorithm with state-of-the-art Job-Task based algorithms. The experimental results validate the efficiency and effectiveness of our proposed algorithm.