基于Docker的Hadoop集群网络性能分析

摘要

针对Hadoop从网络的角度来研究提高Hadoop性能的方法不足的问题,提出了一个通过相关工具,根据不同的网络配置,在虚拟化环境Mininet下监测并分析Hadoop集群的网络特性的系统框架.本系统建立一个在Docker容器内运行的Mininet的Hadoop集群,按照不同的需求,通过配置Mininet方便地修改网络拓扑结构,可以监测容器和虚拟交换机的网络流量,并通过网络流量监视器观察网络行为,分析不同的网络设置对Hadoop性能的影响.实验从集群中没有背景流量、集群中的背景流量无QoS控制、集群中的背景流量有QoS控制,三个方面对Hadoop集群网络性能进行了详细分析.%Aiming at the problems of Hadoop from the perspective of network,this paper proposes a system framework to monitor and analyze the network characteristics of Hadoop cluster in the virtual environment Mininet according to the different network configuration.This system builds a Mininet Hadoop cluster running in the Docker container.According to the different requirements,Mininet can easily modify the network topology,monitor the network traffic of the container and the virtual switch,and observe the network behavior through the network traffic monitor.It analyzed the impact of different network settings on Hadoop performance.The performance of the Hadoop cluster network is analyzed in detail from three aspects:background flow without cluster flow,background traffic without QoS control,and background traffic in the cluster with QoS control.