Research and design of data processing based on ETL framework

摘要

ETL is a key link in the construction of data warehouse. On the base of analyzing the mainstream ETL tool Datastage, the data extraction, transformation and loading, proposes a ETL framework based on data processing, and the realization method and steps are discussed in detail. The framework uses HIVE as a data processing station, improve the operating efficiency of the file;;data task according to the E, T and L three parts and hierarchical partitioning, conversion of data users to better grasp the process;;development data using the configuration file of the task, the development personnel free out from the heavy code, will to shift the focus of the work to the data logical task, which has greatly improved the efficiency of development personnel data processing.