Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm

摘要

In recent years, Storm, an open source distributed real-time computation system, has gained significant amount of popularity in cloud computing industry due to its high reliability and good processing mode. The key in tuning Storm performance lie in the strategy deployed a topology on Storm and the scheduling method used in Storm scheduler. A Storm topology refers to a graph of real-time computation, which provides the logic view of the data process. Currently, Storm adopts a static topology deployment strategy and a simplistic scheduling method, which not only limits flexibility in topology tuning, but also leads to low efficiency in load balancing among its worker nodes. To this end, a Storm topology dynamic optimization algorithm based on the theory of constraints (STDO-TOC) is proposed to dynamically eliminate the performance bottleneck of the topology. In addition, a real-time scheduling algorithm based on topology and traffic (TS-Storm) is proposed to effectively solve the problem of inter-node load imbalance. Extensive experiment results show that, our newly proposed topology deployment strategy and scheduling method can largely improve performance of Storm in term of better system throughput, shorter average delay and latency, and less inter-node traffic.