Improving MapReduce Performance with Progress and Feedback Based Speculative Execution

摘要

Task stragglers dramatically impede parallel job execution of data-intensive computing in Cloud Datacenters Due to the uneven distribution of input data resulted from heterogeneous data nodes, resource contention situations, and network configurations, it causes delay failures due to the violation of job completion time. However, data-intensive computing frameworks, such as MapReduce or Hadoop, employ a mechanism called speculative execution to deal with the straggler issue, speculative execution provide limited effectiveness because in many cases straggler identification occurs too late within a job lifecycle. Identifying the straggler and the timing of identifying it is very important for Straggler mitigation in Data-intensive cloud computing. Speculative execution method is a widely adopted as a straggler identification and mitigation scheme but it has certain inherent limitations. In this paper, we strive to make Hadoop more efficient in cloud environments. We present Progress and Feedback based Speculative Execution Algorithm (PFSE), a new Straggler identification scheme to identify the straggler MapReduce tasks based on the feedback information received from completed tasks beside the progress of the currently processing task, our extensive simulation shows that PFSE can outperform the dynamic scheduling techniques like Self-Learning MapReduce scheduler (SLM) and LATE. PFSE can assist in enhancing straggler Identification and mitigation for tolerating late-timing failures within data intensive cloud computing.