Efficient Coflow Scheduling of Multi-Stage Jobs with Isolation Guarantee

摘要

Coflow scheduling is critical for data-parallel computing performance in datacenters. Performance and isolation guarantee have become two major objectives for coflow scheduling. However, in the context of multi-stage jobs, existing coflow scheduling frameworks only focus on minimizing the average job completion time (JCT) while overlooking the isolation guarantee. To address this problem, in this paper we propose the first coflow scheduling scheme that aims to achieve both objectives. We show that our scheduler outperforms existing alternatives significantly in minimizing average JCT while guaranteeing that no job will be delayed beyond a constant time than its JCT in a fair scheme. Our evaluation results show that our scheduler reduces the average JCT by at least 86% compared with state-of-the-art schedulers.