基于日志分析的增量数据捕获方法研究

摘要

Change data can be captured by scanning log files in the database. However, methods proposed in most existing studies have many limitations and can only be applied to certain types of DBMS. Moreover, they have not provided effective ways of eliminating redundant information. This paper, while analyzing these limitations, proposes a universal model based on log analysis for incremental data detection and its net effect processing,and describes the typical procedures for incremental detection,which are log extraction,log analysis and net effect handling. Finally,experiments are also conducted to analyze the net effect handling speed,redundant data compression ratio,network transmission speed and other factors,through which show that the net effect handling helps to decrease the time of network transmission and data update,and improves operating efficiency.%通过扫描数据库日志文件可以捕获变化数据，但已有研究仅适应某种特定类型的数据库管理系统并且缺乏对冗余信息的消除。针对上述不足，提出一个基于日志分析的增量数据检测及其净效应处理通用模型，描述增量检测过程中的通用处理步骤，即日志抽取、日志分析和净效应处理3个模块。通过实验对净效应处理速度、冗余数据压缩率、网络传输速度等因素进行分析，结果表明，净效应处理可以有效地减少数据的网络传输时间和更新时间，提高运行效率。