Optimization and Research of Hadoop Platform Based on FIFO Scheduler

摘要

As the Hadoop Platform is being more extensively applied on Big Date Processing, Distributed Computing, Cloud Computing etc., the greater performance of it is required. This paper focuses on the insufficient thought for task locality of the default FIFO Scheduler by the Hadoop Platform. And through analysis and study on the system's characteristics optimizing strategy of the Hadoop Platform, the paper provides TLI(Task Locality Improvement) Scheduler. According to the probability's threshold level of the task locality, the jobs shall be set & processed to several job queues. As above the threshold level, the FIFO Scheduler shall be adopted, conversely, the TLI Scheduler shall be applied. For the scheduling tasks, they will be locally executed immediately as the local node is idle, Or they will be executed until the local node is idle. Therefore, the task locality is optimized and the performance is improved. The experiment proofs the task locality improved to 98.0% and the time performance improved 10.9%.