Queries over Large-Scale Log Data of Hybrid Granularities

摘要

Log data is of great value for operation and maintenance of systems and networks. With log data, system behaviors can be monitored, traced, analyzed, to detect unusual circumstances and identify warnings, for the purpose of taking timely measure. However, nowadays many general systems making queries and retrieval over log data is traditionally based on the full amount of raw data. When the amount of log data increase substantially, it's difficult to execute queries based on the raw data of extremely large scales to meet response time requirements. As log data will not be updated after generated, this paper proposes that data and queries can be preprocessed to form data preprocessing results of different granularities. Queries submitted by users can take advantage of corresponding preprocessing results, to improve the query response time. This paper proposes a model for queries over log data of various granularities. The main work includes the follows (1) develop a query model based on hybrid granularity, which enables a query to execute on various granularities of data sets after preprocessing, (2) analyze and prove the completeness and correctness of the proposed query model based on hybrid granularity, (3) describe the query action based on hybrid granularity model formally and present the algorithm framework of the query transformation, (4) analyze and demonstrate the advantages of efficiency of the proposed query model compared to the original data query model. The proposed solution is used in some practical systems. The results show that this solution can guarantee the correctness of the query results while it is able to improve the responsive efficiency of the query significantly. Preprocessing, (2) analyze and prove the completeness and correctness of the proposed query model based on hybrid granularity, (3) describe the query action based on hybrid-granularity model formally and present the algorithm framework of the query transformation, (4) analyze and demonstrate the advantages in efficiency of the proposed query model compared to the original data query model. The proposed solution is used in some practical systems. The results show that this solution can guarantee the correctness of query results while it is able to improve the responsive efficiency of the query significantly.