DataFoundry Data Warehousing and Integration for Scientific Data Management

摘要

Data warehouses and data marts have been successfully applied to a multitude of commercial business applications as tools for integrating and providing access to data located across an enterprise. Although the need for this capability is as vital in the scientific world as in the business domain, working warehouses in our community are scarce. A primary technical reason for this is that our understanding of the concepts being explored in an evolving scientific domain change constantly, leading to rapid changes in the data representation. When any database providing information to a warehouse changes its format, the warehouse must be updated to reflect these changes, or it will not function properly. The cost of maintaining a warehouse using traditional techniques in this environment is prohibitive. This paper describes ideas for dramatically reducing the amount of work that must be done to keep a warehouse up to date in a dynamic, scientific environment. The ideas are being applied in a prototype warehouse called DataFoundry. DataFoundry, currently in use by structural biologists at LLNL, will eventually support scientists at the Department of Energy's Joint Genome Institute.