METHOD FOR CONSTRUCTING AND UTILIZING INDEX TO IMPROVE DATA PROCESSING PERFORMANCE BASED ON MAPREDUCE IN HADOOP ENVIRONMENT

摘要

The present invention relates to a method for processing data based on MapReduce. Specifically, the present invention relates to a method for constructing and utilizing an index to improve the data processing performance based on MapReduce in a Hadoop environment, which constructs a secondary index to effectively process big data with a MapReduce method in the Hadoop environment, and utilizes the secondary index for MapReduce-based data processing. The method for constructing an index comprises the following steps. Each mapper for processing file splits outputs an intermediate result value to be transmitted to a reducer by using an offset, a length, and a key value (K). Then, the reducer calculates the total offset and the total length by finding the smallest offset section and the largest offset section in a list of each of record sections, and stores the total offset and the total length in a split-level Hadoop index file.;COPYRIGHT KIPO 2017