An Investigation of Hadoop Parameters in SDN-enabled Clusters

摘要

Apache Hadoop is an open source framework for distributed and parallel processing of big data jobs. It has its own distributed file system which facilitates local storage and processing on commodity hardware. Hadoop distributed file system is a core part of the Hadoop ecosystem which comprises of large number of configuration parameters. Customizing these parameters to enhance the throughput of the system, for a particular job, may require a lot of experience and skills. During the execution of a Hadoop job in a multi-node cluster, the communication among nodes takes place through switch. These switches have vendor-specific protocols to direct the flow of traffic. Software defined networking has made it possible to make networks more programmable and configurable. In this paper, we analysed the impact of Hadoop distributed file system parameters, like block size, replication factor, MapReduce parameter like number of mapper, and Hive query structure. We used faucet, an OpenFlow switch, to monitor the transfer of both packets in/out of the system to see whether network traffic information can be used to predict the impact of Hadoop parameters. We have also monitored CPU usage, disk usage, memory usage and the overall execution time during the execution of Hadoop jobs. Our investigation showed that customizing these configuration parameters of Hadoop does have an impact on network, system and execution time.