Model-driven scheduling for distributed stream processing systems

摘要

Distributed Stream Processing Systems (DSPS) are “Fast Data” platforms that allow streaming applications to be composed and executed with low latency on commodity clusters and Clouds. Such applications are composed as a Directed Acyclic Graph (DAG) of tasks, with data parallel execution using concurrent task threads on distributed resource slots. Scheduling such DAGs for DSPS has two parts—allocationof threads and resources for a DAG, andmappingthreads to resources. Existing schedulers often address just one of these, make the assumption that performance linearly scales, or usead hocempirical tuning at runtime. Instead, we propose model-driven techniques for both mapping and allocation that rely on low-overheada prioriperformance modeling of tasks. Our scheduling algorithms are able to offer predictable and low resource needs that is suitable for elastic pay-as-you-go Cloud resources, support a high input rate through high VM utilization, and can be combined with other mapping approaches as well. These are validated for micro and application benchmarks, and compared with contemporary schedulers, for the Apache Storm DSPS.