Dynamic partitioning of instances in distributed streaming platform for real-time applications

摘要

Software for a distributed streaming platform monitors a first operator instance in a container for an application running on a distributed streaming platform over a networked cluster of servers. The first operator instance includes an input port that receives streaming windows from an upstream instance and an output port that emits streaming windows to a downstream instance. And the monitoring includes determining a window identifier for a streaming window which the first operator instance is processing. The software reports results of the monitoring to a master and receives a dynamic adjustment from the master. The dynamic adjustment includes launching a second operator instance of the same operator in the container. The software resumes a flow of streaming windows into the input port and the output port according to first recovery policy for the first operator instance and a second recovery policy for the second operator instance.