ESTABLISHING DATA PIPELINES WITH INCLUSION OF ITERATIVE SUB-PIPELINES

摘要

Improved data pipelines are provided. A request to activate a first pipeline based on a first pipeline definition is received. The first pipeline is generated based on the first pipeline definition, wherein generating the first pipeline includes generating a reusable processing module at a first position within the first pipeline, and generating a downstream processing module at a second position within the first pipeline, where the downstream processing module selectively sends output data to the reusable processing module, such that data processed by the first data pipeline will be processed by the identified reusable processing module at least twice. The first pipeline is activated.