Log-based change data capture from schema-free document stores using MapReduce

摘要

Change data capture (CDC) is an approach to data integration that is used to determine and track the data that has changed so that action can be taken using the change data. However, the state of art of change data capture (CDC) in the context of document-oriented NoSQL databases is not mature. Therefore, it is urgent to require a NoSQL CDC solution. Although some manufacturers of NoSQL databases start to research on CDC for NoSQL, these approaches are just for the specific product. In our paper, we propose a log-based CDC approach from abstract schema-free document stores using MapReduce. The process is divided into map and reduce procedures, benefited from MapReduce framework, to generate cell state models (CSMs). In order to infinitely look back to any revision, we enable our proposed CSM to support copy-modify-merge model to manage the revisions of change data. Finally, experimental results show that this approach is independent and appropriate for document stores, with high performance and throughput capacity.