Performance Optimization System for Hadoop and Spark Frameworks

摘要

The optimization of large-scale data sets depends on the technologies andmethods used. The MapReduce model, implemented on Apache Hadoop or Spark,allows splitting large data sets into a set of blocks distributed on several machines.Data compression reduces data size and transfer time between disks and memory butrequires additional processing. Therefore, finding an optimal tradeoff is a challenge,as a high compression factor may underload Input/Output but overload theprocessor. The paper aims to present a system enabling the selection of thecompression tools and tuning the compression factor to reach the best performancein Apache Hadoop and Spark infrastructures based on simulation analyzes.