Column Access-aware In-stream Data Cache with Stream Processing Framework

摘要

In recent years, researches focus on addressing the query bottleneck issue of big data, e.g. NoSQL databases, MapReduce and big data processing framework. Although NoSQL databases have many advantages on On-Line Analytical Processing (OLAP), it is a big project to migrate Relational Database Management System (RDBMS) to NoSQL. Therefore, the optimization of RDBMS is still important. In this paper, we construct Column Access-aware In-stream Data Cache (CAIDC) for relational databases, which is an integral part of RDBMS and in-memory cache. Furthermore, a live synchronization approach from physical RDBMS to in-memory data cache using stream processing framework is proposed. On one hand, CAIDC provides low latency while supporting log-based trigger in the presence of updates to maintain data consistency because of stream processing framework. On the other hand, CAIDC translates the frequently accessed data to column-oriented in-memory cache by the column access frequency to ensure heavy hitter queries. Finally, experimental results show that this approach is supporting a wide range of applications with big data.