Informing, evaluating and automating the record linkage process for reliably combining disparate datasets.

摘要

Linkage is the process of determining whether two records belong to the same person by measuring the similarity of the demographic information tied to each record. It is core to maintaining hospital patient lists, enabling information exchange, and combining datasets for research. Probabilistic linkage, a statistical approach that takes into account the reliability and discrimination of demographic values, has become the dominant linkage method. Like many statistical methods, the performance of probabilistic linkage depends on data preparation, estimation of parameters, and properly fitting the problem to the method---all steps that require human intervention. Because of the time and cost required as well as the variability introduced by humans at each of these steps, linkage is still an active area of research.;A qualitative analysis of the linkage between an enterprise data warehouse and a population database was conducted to compare the frequency of demographic values in the two datasets with a set of records identified as potential duplicates. Doing so allowed patterns to be identified where some frequencies deviated from what was expected and facilitated the creation of a list of simple tools and recommendations that can be used to ensure reliability when undertaking a record linkage project.;An extension of probabilistic record linkage is introduced that preserves the statistical foundation of the method and uses additional information available from demographic values that partially match due to misspellings or typographical errors. Use of this extension results in a 25% reduction in misclassified records and out performs the traditional method regardless of which demographics fields are used for comparison or which cutoff is used for partial matches.;The final study conducted was a quantitative analysis of the impact that data quality and completeness have on record linkage. Using a set of records known to exist in two datasets, this analysis enabled the cause of missed record matches to be determined.;This work investigates how the set of records that are determined to match is affected by characteristics of the datasets, proposes methods to simplify and further automate parameter estimation, and explores how linkage can be evaluated for completeness and accuracy.