Performance Comparison and Analysis of Yarn's Schedulers with Stress Cases

摘要

Hadoop, as a popular distributed storage and computing platform, has been widely used in many companies. Yarn is the resource management platform in Hadoop and plays an important role in the resource managing, because it can affect the cluster's energy efficiency and the usability for applications. The schedulers are the brain of Yarn, which manage and schedule resources from cluster to applications. In this paper, we conduct experiments to compare and analyze the performance of Yarn's schedulers. We use various scenarios to demonstrate the strengths and weaknesses of each scheduler from the perspective of response speed, cluster's efficiency, scheduler's speciality etc. Experimental results demonstrate that the FIFO Scheduler has a better performance and data locality sense for batch jobs processing than the other schedulers, but the Capacity Scheduler and the FIFO Scheduler have better response speed and cluster's usability than the FIFO Scheduler which has a hunger problem in mixed scenario.