粒子群算法在分布式 ETL 任务调度中的应用

摘要

随着分布式数据环境越来越复杂,ETL 工具要面临数据源多、分布地域广和海量数据等因素带来的挑战。原有的集中式 ETL 工作流优化理论不能满足现在复杂数据环境的要求。介绍了如何将基于置换的离散型粒子群算法应用到分布式 ETL 任务优化调度问题上,主要工作围绕 ETL 工作调度模型、算法编码设计、目标函数选择等内容来展开,给出了分布式 ETL 工作调度策略的实现过程和伪代码。理论分析和实验证明了实际应用的有效可行性。%With the increasing complexity of distributed data environment, ETL tools face the challenge of many data sources, geographic distribution, massive data and other factors. The original centralized ETL workflow optimization theory can not meet the demands of the environment of the complex data. This paper presents how the discrete particle swarm optimization based on replacement is used in task scheduling of the distributed ETL. The main contents include the abstraction of the ETL task schedul-ing model, design of the algorithm coding, selection of objective function and so on. The realization and pseudocode of distributed ETL job scheduling strategy are also mentioned. The theory and experiment prove it to be feasible and efficient.