Enhancing network I/o performance for a virtualized Hadooprncluster

摘要

A MapReduce programming model is proposed to process big data using Hadoop, one of thernmajor cloud computing frameworks. With the increasing adoption of cloud computing, runningrna Hadoop framework on a virtualized cluster is a compelling approach to reducing costs andrnincreasing efficiency. In this paper, we measure the performance of a virtualized network andrnanalyze the impact of network performance on Hadoop workloads running on a virtualized cluster.rnThen, we propose a virtualized network I/O architecture as a novel optimization for arnvirtualized Hadoop cluster for a public/private cloud provider. The proposed network architecturerncombines traditional network configurations and achieves better performance for Hadooprnworkloads. We also show a better way to utilize the rack awareness feature of the Hadooprnframework in the proposed computing environment. The evaluation demonstrates that the proposedrnnetwork architecture and mechanisms improve performance by up to 4.1 times comparedrnwith a bridge network architecture. This novel architecture can even virtually match the performancernof the expensive, hardware‐based single root I/O virtualization network architecture.