Managing a complicated workflow based on dataflow-based workflow scheduler

摘要

Diversifying requirements forces a data analytics platform to deal with complicated workflows including various combinations of data integration. We have developed a new workflow scheduler for our Hadoop-based data analytics platform to cope with the issue. The new scheduler enables fine-grained dependency management efficiently on a huge monolithic workflow and can improve the utilization efficiency of computational resources while keeping the maintainability of the workflow. This paper describes the design of the new workflow scheduler and introduces how we have been operating the data analytics platform based on the scheduler.