Data Management for the RedisDG Scientific Workflow Engine

摘要

In this paper we investigate the general problem of controlling a scientic workflow service in termsof data management. We focus on the data managementproblem for the RedisDG scientific workflow engine.RedisDG is based on the Publish/Subscribe paradigmfor the interaction between the different componentsof the system, hence new issues appear for scheduling. Indeed, the Publish/Subscribe paradigm utilization introduces different challenging problems, among them the design of effective solutions for managing data, on the fly, when tasks are published. Our contributions are twofold. First we add new functionalities to the RedisDG workflow engine with scheduling decisions related to the allocation of data intensive jobs to compute units and according to an efficient management of data and second we introduce a large set of experiments tovalidate our approaches. We analyze our results and wealso sketch perspectives and insights. Experiments areconducted on the Grid'5000 testbed and the paper isa step forward to implement a 'Workflow engine as aService' (WaaS).