The massively growing data volume and the pressing need for low latency are pushing the traditional store-first-query-later data warehousing technologies beyond their limits. Many enterprise applications are now based on continuous analytics of data streams. While integrating stream processing with query processing takes advantage of SQL's expressive power and DBMS's data management capability, it raises serious challenges in dealing with complex dataflow, applying queries to unbounded stream data, and providing highly scalable, dynamically configurable, elastic infrastructure. To solve these problems, we model the general graph-structured, continuous dataflow analytics as a SQL Streaming Process with multiple connected and stationed continuous queries; then we extend the query engine to support cycle-based query execution for processing unbounded stream data chunk-wise with sound semantics; and finally, we develop the Query Engine Net (QE-Net) over the Distributed Caching Platforms (DCP) as a dynamically configurable elastic infrastructure for parallel and distributed execution of SQL Streaming Processes. We extended the PostgreSQL engines for building the QE-Net infrastructure. Our experience shows its merit in leveraging SQL and query processing to analyze real-time, graph-structured and unbounded streams. Integrating it with a commercial and proprietary MPP based database cluster is being investigated.
展开▼
