A Metadata Diagnostic Framework for a New Approximate Query Engine Working with Granulated Data Summaries

摘要

This paper refers to a new database engine that acquires and utilizes granulated data summaries for the purposes of fast approximate execution of analytical SQL statements. We focus on the task of creation of a relational metadata repository which enables the engine developers and users to investigate the collected data summaries independently from the engine itself. We discuss how the design of the considered repository evolved over time from both conceptual and software engineering perspectives, addressing the challenges of conversion and accessibility of the internal engine contents that can represent hundreds of terabytes of the original data. We show some scenarios of a usage of the obtained metadata repository for both diagnostic and analytical purposes. We pay a particular attention to the relationships of the discussed scenarios with the principles of rough sets - one of the theories that hugely influenced the presented solutions. We also report some empirical results obtained for relatively small fragments (100 × 2~(16) rows each) of data sets coming from two organizations that use the considered new engine.