基于多变量动态算法的Hadoop负载均衡优化与实现

摘要

Cloud computing system and Hadoop load balance mechanism were introduced particularly.When Hadoop system needs load balance, the user must call the system's balancer with the threshold being input in advance which restrict the performance of the balancer greatly.In this case, a multivariable dynamic evaluate threshold algorithm was used to optimize and realize the balancer.Multivariable dynamic evaluate threshold algorithm is one of the algorithm based on the current state of the cluster.It used the disk space utilization and cluster busy degree to dynamic evaluates the threshold of balancer.This algorithm not only can dynamic evaluates the balancer's threshold, but also can balance the desired balanced result and the running time by means of selection suitable weight parameters.Results of the experiment showed that comparing with the system default balancer, which needed a fixed threshold.The balancer using multivariable dynamic evaluate threshold algorithm would be more satisfactory.%分析了云计算平台Hadoop负载均衡机制,针对Hadoop系统自带的均衡器要预先输入阈值才能运行的不足,通过多变量动态估计阈值对其进行优化与实现,依据集群磁盘空间负载情况和集群繁忙程度,动态估计均衡器的阈值,并给出了与估计阈值相关的计算公式,在此基础上采用集群系统对上述算法进行了测试验证.实验结果表明,对比Hadoop系统自带的需要固定阈值的均衡器,新方法能在均衡过程中利用系统相关信息动态估计阈值,对Hadoop的负载均衡优化取得更加令人满意的效果.