Smart Partitioning Mechanism for Dealing with Intermediate Data Skew in Reduce Task on Cloud Computing

摘要

MapReduce greatly alleviates the burdens of programmers and gradually becomes an application programming standard on cloud computing nowadays, because the run-time system of cloud computing can automatically handle the issues of paralleled and distributed programming on behalf of programmers at run time. Although MapReduce can strongly benefit programmers on developing cloud computing applications, intermediate data skew inevitably hurts application performances. MapReduce can use the Smart Partitioning Mechanism (SPM) proposed in this paper as an alternative solution to deal with intermediate data skew in Reduce tasks on cloud computing. With the capability of averagely distributing intermediate data over Slave nodes in SPM, MapReduce no longer suffers from the performance penalty resulting from the intermediate data skew problem in Reduce tasks on cloud computing.