Formula-based load evaluation in distributed streaming platform for real-time applications

摘要

Software for a distributed streaming platform receives an application that runs on a streaming platform. The application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes. The application is associated with a pre-defined hint that is a key-value pair. The software launches the application by assigning the instances of operators to containers provided by the streaming platform and initiating the streams. Then the software reads a value for the pre-defined hint and transmits the value to the application through an application programming interface (API) exposed by the streaming platform. The software receives a request from the application through the API to make a dynamic adjustment. And the software makes the dynamic adjustment and re-launches the application using a recovery policy.