L-Heron: An open-source load-aware online scheduler for Apache Heron

摘要

Apache Heron has emerged as a promising Data Stream Processing System (DSPS). However, it lacks intelligent scheduling strategy, which results in significant performance degradation for streaming applications in certain scenarios. In this paper, we first illustrate the inefficiencies and challenges of Heron default scheduling in current practice through experimental observations and analysis. Motivated by our observations, we propose L-Heron, an online scheduler based on Heron, which has the following features: (i) based on runtime information, it can improve the data processing efficiency by using the load-aware online scheduling, which heuristically minimizes the overall communication overhead by identifying the traffic load; (ii) it is load aware, which can effectively balance the workload of a topology to avoid heavy performance loss caused by overloading of worker nodes; (iii) it provides an online scheduling interface that is transparent to users, which allows users to focus on their scheduling logic and easily deploy them to the system. Additionally, we have evaluated L-Heron on well-known example topologies and a realistic application. Extensive experimental results show that the effectiveness of L-Heron is consistent among multiple metrics including the system completion latency, inter-node traffic, CPU utilization and throughput, with respect to Heron and recent related work.