Parallelization of Massive Multiway Stream Joins on Manycore CPUs

摘要

Joining a high number of data streams efficiently in terms of required memory and CPU time still poses a challenge. While binary join trees are very common in database systems, they are mostly unusable for streaming queries with tight latency constraints when the number of streaming sources is increasing. Multiway stream joins, on the other hand, are very suitable for this task since they are mostly independent of the non-optimal ordering of join operators or huge intermediate join results. In this paper, we discuss challenges but also opportunities for multi-way stream joins for modern hardware, especially manycore processors. We describe different parallelization and optimization strategies to allow a streaming query to join up to 256 streams on a single CPU while keeping individual tuple response time and also memory footprint low. Our results show that a multiway join can perform magnitudes faster than a binary join tree. In addition, further tuning for efficient parallelism can improve performance again for a factor up to a magnitude.