MIGRATING A MULTITERABYTE ARCHIVE FROM OBJECT TO RELATIONAL DATABASES

摘要

A commercial, object-oriented database engine with custom tools for data-mining the multiterabyte Sloan Digital Sky Survey archive did not meet its performance objectives. We describe the problems, technical issues, and process of migrating this large data set project to relational database technology. The advent of digital archives―enabled by quantum leaps in the technology to publish, distribute, and mine data over the Internet―has created a science and engineering data avalanche. The Human Genome Project (www.ornl.gov/ TechResources/Human_Genome/home.html) and the CERN Large Hadron Collider (http:// lhc-new-homepage.web.cern.ch/lhc-new-homepage) are two examples of very large scientific data sets coming online in the biological and particle physics communities, respectively. Astronomy is no exception, and soon might be deluged with more new data from current and proposed sky surveys than any other science. In this age of multiterabyte scientific archives, scientists need to share common lessons from different disciplines to make the most of available opportunities and to avoid being overwhelmed by the flood of data.