A Heuristic Scheduling Approach to Hybrid Makepsan Problem for Data Intensive Computing

摘要

Data intensive computing (DIC) offers an attractive option for business to remotely execute applications and load the computing resources from cloud in a streaming way. A key challenge in such environment is to increase the utilization of cloud cluster for the high throughput processing. One way of achieving this goal is to optimize the execution of computing jobs on the cluster. We observe that the order in which these jobs are executed can have a significant impact on their overall completion time (makespan). Our goal is to design a job scheduler that minimizes the makespan. In this study, a new formalization is introduced to present each job as a pair of disk processing and network transmitting two-stage durations. Due to the streaming processing feature, the two-stage operations are executed in an overlap manner and may lead to both one-stage and two-stage scheduling situations. A novel heuristic scheduling strategy is proposed for this hybrid scheduling problem, and the performance of the method is confirmed by the experimental evaluation.