Graph partition–based data and task co-scheduling of scientific workflow in geo-distributed datacenters

摘要

Most large-scale scientific workflows take place in multiple collaborative datacenters for accessto community-wide resources, while adhering to each datacenter's non-uniform resource limits.However, moving both initial input datasets with predetermined locations and intermediatedatasets needing placement decisions across geo-distributed datacenters hinders efficientexecution of large-scale data-intensive scientific workflows. Thus, scientific workflow's data andtask co-scheduling deal with situations such as pre-placed initial input datasets, placement ofintermediate datasets and each datacenter's non-uniform computation and storage constraint,while minimizing the cross-datacenter data transfer. Since this scheduling problem is known tobe NP-hard, here, we propose a novel approach, based on the multilevel graph coarsening anduncoarsening framework, togetherwith a specialized hybrid genetic algorithm having distinctivegraph partition driven features of repair and local improvement, for scheduling data-intensivescientific workflows in geo-distributed datacenters and optimizing the cross-datacenter datatransfer volume. Extensive simulations, based on four real-world workflow traces, show thatour algorithm significantly reduces the overall geo-distributed data transfer and demonstrate itseffectiveness.