DEVICE AND METHOD FOR OPTIMIZATION OF DATA PROCESSING IN MapReduce FRAMEWORK

摘要

PROBLEM TO BE SOLVED: To provide a device and method for optimization of data processing in a MapReduce framework.SOLUTION: A map reduce frame work for large scale data processing is optimized by a method of the invention that can be implemented by a master node. The method comprises receiving from worker nodes data on read pointer locations pointing to input data of tasks executed by the worker nodes, and stealing work from these tasks. The work is applied to input data that have not yet been processed by the task from which the work is stolen.