FastFlow: Efficient Scalable Model-Driven Framework for Processing Massive Mobile Stream Data

摘要

Massive stream data mining and computing require dealing with an infinite sequence of data items with low latency. As far as we know, current Stream Processing Engines (SPEs) cannot handle massive stream data efficiently due to their inability of horizontal computation modeling and lack of interactive query. In this paper, we detail the challenges of stream data processing and introduce FastFlow, a model-driven infrastructure. FastFlow differs from other existing SPEs in terms of its user-friendly interface, support of complex operators, heterogeneous outputs, extensible computing model, and real-time deployment. Further, FastFlow includes optimizers to reorganize the execution topology for batch query to reduce resource cost rather than executing each query independently.