基于大数据处理的ETL框架的研究与设计

摘要

针对传统抽取、转换、装载（ETL）架构在数据处理过程控制方面和数据产品开发效率方面的不足，提出一种基于大数据处理的ETL架构。通过分析主流的ETL工具--Datastage的工作原理和ETL过程的特点，设计ETL元数据描述模块、ETL任务描述模块、配置解析模块和数据任务调度模块等。使用该框架处理数据任务时，以配置文件的方式开发，使工作效率得到了极大的提升。基于该设计思想开发一款基于大数据处理的ETL工具，在对数据处理过程（E、T、L）的控制方面得到了改进，同时也可以使数据开发人员从大量重复的操作中解脱出来，将更多的精力放在数据的逻辑处理方面。%According to the traditional extraction, conversion, loading (ETL) architecture for process control deficiencies in data processing and data product development efficiency, the article presents a data processing based on ETL. The characteristics of ETL working principle and process analysis of the mainstream ETL tool--Datastage, design of ETL metadata description module, ETL module, configuration task description analysis module and data task scheduling module etc. Using the framework of data processing tasks, development to the configuration file of the way, so that the work efficiency has been greatly improved. The design idea of the development of a large data processing based on ETL tools, in data processing (E,T,L) of the control has been improved, but also can make the data developer freed from a large number of repetitive operations, will focus more on the aspects of the data logic processing.