SHUFFLE REDUCE TASKS TO REDUCE I/O OVERHEAD

摘要

A Shuffle Reduce operation receives as input files that have been sorted and written by different map tasks, fetches batches of data from each input file, and merges and sorts the batches of data to form a large unified piece of data. A Shuffle Reduce operation is applied to the unified piece of data to produce output data. The Shuffle Reduce operation includes a commutative reduce operation that provides an amount of output data that is significantly less than an amount of input data. The output data is written to memory. The process is repeated for different batches of data until data from each input file is entirely consumed and the output data has been fully formed. The Shuffle Reduce operation greatly reduces the data size that needs to be read by the reduce tasks in a Shuffle operation, thereby significantly reducing the input/output overhead and total execution time.