Efficient entity resolution using multiple blocking keys for bibliographic dataset

摘要

Entity Resolution(ER) is defined as identifying entities referring to the same real world object. The standard entity resolution process compare each entity with all other entities, which is inefficient for large datasets. A significant challenge in ER is to reduce the search space and execution time. The aim of this paper is to provide efficient entity resolution implementation for massive dataset by combining the use of multiple blocking key and parallel and distributed computing. In multiple blocking key concept, a record can belong to multiple blocks and it is possible that a record pair is generated multiple times for matching task. A solution to eliminate these duplicate pair is proposed, in addition to this character based similarity measure on sorted tokens is used for computing similarity between two record in the matching task. Efficient partitioning technique is used to remove the limitations of skewed dataset and matching task are evenly distributed among all the reducer. We used a bibliographic dataset in our experiment to show that our approach is less time consuming and scalable.