Performance analysis of matrix and graph computations using data compression techniques in mpi and hadoop mapreduce in big data framework

摘要

In High Performance Computing (HPC) or High Throughput Computing (HTC) applications, matrix and graph computations need huge memory requirements. The data compression techniques and Hadoop implementation of MapReduce have been used for HPC or HTC applications. The data storage, processing time and data compression techniques are required for the matrix and graph computations to understand the performance and scalability analysis. This paper presents the designing and implementation of a Network Overlapped Compression (NOC) theme and Compression Aware Storage (CAS) theme. The working of these techniques reduces information load time and hides compression overhead by interleaving network input-output transfer with compression. The process of compression reduces the quantity of task correspondence and creates uneven work distribution. The MapReduce parallel programming paradigm ought to alleviate quantitative relation. The designed MapReduce Module acknowledges the characteristics of compressed information to boost resource allocation and cargo balance, jointly, NOC, CAS and MapReduce Module decrease job execution time on the average by 66% and information load time by 31%.