Achieving High Freshness and Optimal Throughput in CPU-Limited Execution of Multi-join Continuous Queries

摘要

Due to high data volumes and unpredictable arrival rates, continuous query systems processing expensive queries in real-time may fail to keep up with the input data streams - resulting in buffer overflow and uncontrolled data loss. We explore join direction adaptation (JDA) to tackle CPU-limited processing of multi-join stream queries. The existing JDA solutions allocate the scarce CPU resources to the most productive half-way join within a single operator. We instead leverage the operator interdependencies to optimize the overall query throughput. We identify result stateness, typically ignored by most state-of-the-art techniques, as a critical issue in CPU-limited processing. It gets further aggravated if throughput optimizing techniques are employed. We establish the novel path-productivity model and the Freshness predicate. Our proposed JAQPOT approach is the first integrated solution to achieve near optimal query throughput while also guaranteeing freshness satisfiability. JAQPOT runs in quadratic time of the number of streams irrespective of the query plan shape. Our experimental study demonstrates the superiority of JAQPOT in achieving higher throughput than the state-of-the-art JDA strategy while also fulfilling freshness predicates.