Dynamic mirror based real-time query contention solution for support big real-time data analysis

摘要

Real-time data warehouse is one of the important research field in the data management. Real-time data query and import can bring about the problem of query contention. Query contention will not only seriously affect the accuracy of query analysis, but also reduce the performance of the real-time data warehouse. In this paper, combining an external dynamic storage area, a dynamic mirror replication technology is proposed to avoid the contention between OLAP queries and OLTP updates. An external dynamic storage area (DSA) for real-time data warehouse is established based on the TPC-H benchmark, and an improved ETL framework is proposed to deal with the real-time data and historical data, respectively. The experimental results demonstrate that the proposed solution can get better performance in terms of the effectiveness.