System and method for tracking flow of data during map-reduce job execution in hadoop

摘要

Disclosed is a method and system for tracking flow of data in a distributed file system. The system may receive a MapReduce job application at first. Subsequently, the system may identify relevant locations of code of the MapReduce job application. The system may instrument the code by adding one or more program statements at the relevant locations of the code. The instrumented code may be executed at each node to process big data. The system may receive processing details of the big data from each node. The system may aggregate the processing details to generate a hierarchical dataflow map to be used for tracking flow of the data in the distributed file system.