Improving Speculative Execution Performance with Coworker for Cloud Computing

摘要

MapReduce is an important programming model for large-scale parallel applications. It divides a job into several parallel tasks and completes the job by sequential phases, i.e. map phase and reduce phase. The job completion time will be delayed when a task, called straggler, consumes more time than others. The main reason that a straggler occurs is the imbalance resource distribution among computing nodes in the cloud. Speculative execution is a solution for dealing with stragglers. Duplicate tasks are launched on other nodes to process the same data as the straggler does. Any completion of these tasks implies that this task is finished and other duplicate tasks can be aborted. However, aborting tasks misspends resources. In this paper, we propose an idea of using coworkers to help a straggler. According to the processing rate of the straggler and the coworker, the amount of data parceled out from the straggler to the coworker should be determined. Different from speculative execution, coworkers finish tasks with stragglers and do not misspend computing resources. Experimental results show that coworkers can reduce the task completion time by 37% and the network traffic by 64% when comparing with speculative execution.