DISTRIBUTED JOB SCHEDULER WITH INTELLIGENT JOB SPLITTING

摘要

Methods and systems for improving the performance of a distributed job scheduler by dynamically splitting and distributing the work of a single job into parallelizable tasks that are executed among multiple nodes in a cluster are described. The distributed job scheduler may split a job into a plurality of tasks and assign the tasks to nodes within the cluster based on a time remaining to complete the job, an estimated time to complete the job, and a number of identified healthy nodes within the cluster. The distributed job scheduler may monitor job progress over time and adjust (e.g., increase) the number of nodes used to execute the plurality of tasks if the time remaining to complete the job falls below a threshold amount of time or if the time remaining to complete the job minus the estimated time to complete the job falls below the threshold amount of time.