数据集成增量检测中Sort Merge算法的改进和实现

摘要

在数据集成应用中,增量检测一个最主要的职责就是检测计算出数据源的增量数据,有多种方式可以检测分析出数据源的变化数据:触发器、日志分析、快照差分等等.而快照差分又可以采用不同的算法对两个快照进行计算,针对差分算法中SortMerge算法的一个改进和实现,主要利用有序文件来提高增量计算的效率.由于快照差分在字符串比对、文件差异计算、大数据量的差异对比、自然连接等实际问题有应用,对此类问题都有一定借鉴意义.%In the application of data integration, the main duty of increment detection is to detect and calculate the incremental data of data source.There are various methods to be used to detect and analyse the varied data of data source: the trigger, the log analyst, and the snapshot difference, etc.And for the snapshot difference, different algorithms can be used to calculate two snapshots.This paper focuses on improving and implementing one of those algorithms, the Sortmerge algorithm, and the efficiency of increments computation is improved mainly through the sorted file.Since the snapshot difference is applied in practical problems including the string comparison, the file difference calculation, the mass data difference comparison and the natural join, etc., it provides the reference to problems of similar kinds to a certain extent.