SYSTEM FOR DATA ENGINEERING AND DATA SCIENCE PROCESS MANAGEMENT

摘要

Big data platform for data processing. An exemplary system for managing data engineering and data science processes referenced herein includes an input application module configured to read data inputs from data sources, a processing module configured to apply functions of data science and data engineering processing on the data inputs, a storage module configured to store data inputs, processed data, and output data, an output application module configured to collect the processed data and writes data outputs, an orchestrator module configured to manage the dataflow with predefined rules on which modules to be triggered in accordance with the data inputs and data outputs, and a messaging module configured to communicate the processing module and the orchestrator module.