Implementation of on-process aggregation for efficient big data processing in Hadoop MapReduce environment

摘要

The term Big Data, refers to sizably voluminous data whose volume, variability, and velocity make it very arduous to manage, process or analyzed. To analyze this sizably voluminous kind of data Hadoop will be utilized. However, Processing is very time-consuming. To resolve this quandary & to decrement replication time one solution is to executing the job partially, where an approximate, early result becomes available to the utilizer, afore completion of job. Proposed system gives a more incipient MapReduce architecture that sanctions data to be divided for easier & early processing. This is not time consuming and amends system utilization for batch jobs as well. Proposed system presents a more incipient version of the Hadoop MapReduce framework that fortifies on-Process aggregation, which sanctions & avails users to get early results of a job as it is computing. It will evaluate this technique utilizing authentic-world datasets and applications and endeavor to amend the systems performance in terms of precision and time. Also the combiner introduced in this system is local reducer. Combiner will get execute after map function & before reducer. Instead of processing complete file on-process aggregation divides the file into number of blocks which helps to gives the result in slots. Dividing the file into number of data sets helps to give result as early as possible by giving intermediate result to the user. The objective of the proposed technique is to amend the performance of Hadoop MapReduce for efficient & easy Immensely Big Data Processing time.