改进的Spark Shuffle内存分配算法

摘要

Shuffle性能是影响大数据集群性能的重要指标,Spark自身的Shuffle内存分配算法试图为内存池中的每一个Task平均分配内存,但是在实验中发现,由于各Task对于内存需求的不均衡导致了内存的浪费和运行效率较低的问题.针对上述问题,提出一种改进的Spark Shuffle内存分配算法.该算法根据Task的内存申请量和历史运行数据将Task按内存需求分为大小两类,对小内存需求型Task作“分割化”处理,对大内存需求型Task基于Task溢出次数和溢出后等待时间分配内存.该算法充分利用内存池的空闲内存,可以在数据倾斜导致的Task内存需求不均衡的情况下进行Task内存分配的自适应调节.实验结果表明,改进后算法较原算法降低了Task的溢出率,减少了Task的周转时间,提高了集群的运行性能.%Shuffle performance is an important indicator of affecting cluster pedormance for big data frameworks.The Shuffle memory allocation algorithm of Spark itself tries to allocate memory evenly for every Task in the memory pool,but it is found in experiments that the memory was wasted and the efficiency was low due to the imbalance of memory requirements for each Task.In order to solve the problem,an improved Spark Shuffle memory allocation algorithm was proposed.According to the amount of memory applications and historical operating data,the Task was divided into two categories based on memory requirements.The "split" processing was carried out for the Task of small memory requirements,while the memory was allocated based on the number of Task overflows and the waiting time after overflow for the Task of large memory requirements.By taking full advantage of the free memory of memory pool,the adaptive adjustment of Task memory allocation could be realized under the condition of unbalanced Task memory requirements caused by the data skew.The experimental results show that,compared with the original algorithm,the improved algorithm can reduce the overflow rate of the Task,decrease the turnaround time of the Task,and improve the running performance of the cluster.