System and method for efficient task scheduling in heterogeneous, distributed compute infrastructures via pervasive diagnosis

摘要

A system and method schedules jobs in a cluster of compute nodes. A job with an unknown resource requirement profile is received. The job includes a plurality of tasks. Execution of some of the plurality of tasks is scheduled on compute nodes of the cluster with differing capability profiles. Timing information regarding execution time of the scheduled tasks is received. A resource requirement profile for the job is inferred based on the received timing information and the differing capability profiles. Execution of remaining tasks of the job is scheduled on the compute nodes of the cluster using the resource requirement profile.