Analysis of Job Scheduling Algorithms and Studying Dynamic Job Ordering to Optimize MapReduce

摘要

As there was a big rise in the Big Data field, Hadoop became one of the most used platforms in many applications like clinical data analysis, Facebook, Amazon, in which Big Data processing and utilization is required. One of the most important features that made Hadoop, one of the most popular platform, is adopting MapReduce, which made large changes in the market by processing huge amount of data in parallel technique by distributing data across multiple TaskTracker nodes and each node splits data by Map slots and shrinks the output of Map slots (key/ value) by Reduce slots. However, different algorithms are used for job scheduling in MapReduce which is used to improve resource utilization, job allocation, and minimizing processing time. However, research is still underway to propose optimal method to improve MapReduce model, and there are still several major drawbacks that are still not well studied. In this paper, we studied and analyzed different job scheduling, job ordering algorithms, and dynamic slot configuration. The study discussed the most popular and efficient systems design to find the most efficient improvements for MapReduce and review the corresponding solutions. In our proposed method, we are applying classification algorithm which is going to classify job into highest utilization and poor utilization. After that, the highest utilization job will be forwarded to PRISM algorithm, which will schedule into phase level as there are variations in processing time and resource requirement in each phase, for different kinds of jobs along with the application of dynamic slot configuration, which helps to improve resources utilization and reduce time.