RB-storm: Resource Balance Scheduling in Apache Storm

摘要

Apache Storm is a distributed stream processing computation framework, which has become a preferred technology for near-realtime big data processing by many organizations worldwide. However, Storm lacks an intelligent scheduling mechanism. The default round-robin scheduling which ignores the imbalance of resource requirements of tasks and resource availability of work nodes can leads to serious resource wastage. In this paper, by defining the Resource Requirement Vector (RRV) of tasks and the Resource Utilization Vector (RUV) of work nodes, we propose a resource balance scheduling algorithm, namely RB-storm, to minimize the resources wastage with the consideration of the load of work nodes. The experimental results show that RB-storm algorithm can reduce resources wastage and ensure the performance of Storm cluster.