Seflow: Efcient Flow Scheduling for Data-Parallel Jobs

摘要

Data-parallel jobs transfer massive amounts of data between a series of successive stages. The coflow abstraction is proposed to represent a group of parallel flows between two stages and efficiently improves stage-level performance. However, state-of-the-art coflow scheduling techniques are agnostic to the jobs' inter-coflow semantics and thus are suboptimal in reducing the average job completion times (JCT). To address this problem, in this paper we present the "semantic flow" (seflow) abstraction to express the job-level intercoflow semantics. A seflow comprises not only all the coflows of a job but also the relationship between the coflows. We design an efficient seflow scheduler which utilizes the rich seflow semantics of jobs to achieve better performance than seflow-agnostic scheduling for data-parallel jobs.