Queries over Large-Scale Incremental Data of Hybrid Granularities

摘要

The development of Internet and Web systems in recent years has made it difficult and challenging to deal with large-scale data. What we usually need to process is incremental data, the scale of which will enlarge with the changing of time. Nowadays, many general queries over data is traditionally based on the full amount of raw data, which becomes a great challenge to performance when the data increases substantially. As these data will not be updated after generated, this paper proposes a query model, called hybrid-granularity model, that data and queries can be preprocessed to form intermediate result sets of different granularities. With query transformation, submitted queries can take advantage of intermediate preprocessing results, to obtain the required final results. In this paper, we also describe query transformation and the methods to seek the solution of best performance with hybrid-granularity model for the specific query. Finally, we analyze and verify the advantage on performance of the proposed model by comparison with the original model in experiment. The proposed solution is used in some practical systems, which shows that this solution can guarantee the correctness of query results while improving the responsive efficiency of the query significantly.