Transform Merging of ETL Data Flow Plan

摘要

ETL (Extract-Transform-Load) is the process of loading a data mart/warehouse. It is often modeled in a data flow Plan in which individual transforms perform various types of operations to convert, cleanse, and integrate dissimilar source data before loading into the target system. Due to the nature of complexity, high-performance process is often hard to achieve. In this paper, we present a transform merging technique to improve the ETL performance. This technique reshapes the transformation plan from the existing transform-based process to a new column-based process where independent column threads can be produced and executed in parallel. The plan reshaping is achieved by plan analysis and transform merge. Our implementation shows that this technique delivers significant performance improvement over the existing execution mechanisms.