Leveraging external user-generated information for large-scale data integration.

摘要

The proliferation of data sources both in the private and public domains (e.g., in enterprise environments and on the World-Wide Web) underscores the need for data integration systems. The purpose of a data integration system is to enable users to access data residing in multiple heterogenous sources through a uniform interface. Manual solutions for building such systems are not a viable option, especially when dealing with large-scale and complex applications.;This dissertation studies the automation of building data integration systems. In particular, it addresses three key challenges that lie at the heart of any such system.;The first challenge relates to the construction of wrappers for the unstructured sources. A source wrapper would ensure that the data in the underlying source is perceived as structured data by the other parts of the system. We particularly focus on sources containing data formatted as lists, and propose a new solution for extracting relational tables from them. The proposed solution is completely unsupervised and domain-independent. It is based on leveraging various sources of information, including a corpus of tens of millions of relational tables published by users on the Web.;The second and third challenges are concerned with establishing semantic mappings across data sources. We first propose a new solution for discovering the correspondences across the elements of two schemas. Then, based on these simple correspondences, we propose another solution to discover more complex declarative mapping rules that can actually be used to transform data and queries across the two schemas. The key underpinning for these two solutions is that, unlike previous approaches, they both exploit the usage information extracted from database query logs. This work is the first to introduce the usage-based approach for establishing mappings across data sources.;To evaluate our approaches, we conducted experiments using realistic data sets, such as real web lists for the wrapper construction work; and schemas and query logs from the retail and life sciences domains for the work on semantic mappings. The experimental results have verified the effectiveness and applicability of our proposed approaches.