Towards optimized scheduling for data-intensive scientific workflow in multiple datacenter environment

摘要

In the big data era, scientific workflow exhibits the characteristics of data intensity and becomesrnincreasingly popular in scientific domains. Efficient scheduling of data-intensive scientific workflow inrna multiple datacenter (DC) environment has been a long-standing challenge. Most of previous work onrndata-intensive scientific workflow scheduling primarily focused on the optimization of reducing the volumesrnof data transfer between workflow tasks. In this paper, novel scheduling strategies for the execution ofrndata-intensive scientific workflow in multi-DC environment are proposed aiming at the optimization of thernoverall data transfer time. A novel DC selection approach is proposed to minimize the number of DCs havingrnenough storage capacity for the execution of scientific workflow as well as optimized inter-DC network bandwidthrnfor efficient data transfer between workflow tasks. A k-means clustering-based data placement strategyrnis adopted to intelligently place the initial data of scientific workflow thereby reducing the volume of initialrndata transfer between different DCs. A multilevel task replication scheduling strategy is invented to reducernthe volumes of intermediate data transfer between DCs during the runtime of the scientific workflow. Simulationsrnspanning a broad range of scientific workflow and multi-DC settings are performed in order to verifyrnthe proposed approaches. The numerical results show that our combined scheduling strategy significantlyrnreduces the overall data transfer time and data transfer volume when scientific workflow is scheduled inrnmulti-DC environment.