A Meta Data Vault Approach for Evolutionary Integration of Big Data Sets : Case Study Using the NCBI Database for Genetic Variation

摘要

A data warehouse integrates data from various and heterogeneous data sources and creates aconsolidated view of the data that is optimized for reporting and analysis. Today, business andtechnology are constantly evolving, which directly affects the data sources. New data sources canemerge while some can become unavailable. The DW or the data mart that is based on these datasources needs to reflect these changes. Various solutions to adapt a data warehouse after the changesin the data sources and the business requirements have been proposed in the literature [1]. However,research in the problem of DW evolution has focused mainly on managing changes in the dimensionalmodel while other aspects related to the ETL, and maintaining the history of changes has not beenaddressed. The paper presents a Meta Data vault model that includes a data vault based datawarehouse and a master data management. A major area of focus in this research is to keep bothhistory of changes and a “single version of the truth,” through an MDM, integrated with the DW. Thepaper also outlines the load patterns used to load data into the data warehouse and materialized viewsto deliver data to end-users. To test the proposed model, we have used big data sets from the biomedicalfield and for each modification of the data source schema, we outline the changes that need to be madeto the EDW, the data marts and the ETL.