Selecting cloud service providers to perform data processing jobs based on a plan for a cloud pipeline including processing stages

摘要

Cloud service providers are selected to perform a data processing job based on information about the cloud service providers and criteria of the job. A plan for a cloud pipeline for performing the job is designed based on the information about the cloud service providers. The plan comprises processing stages each of which indicates processing upon a subset of a data set of the job. Allocated resources of the set of cloud service providers are mapped to the processing stages. Instructions and software images based on the plan are generated. The instructions and the software images implement the cloud pipeline for performing the data processing job. The instructions and the software images are transmitted to machines of the cloud service providers. The machines and the performing of the job are monitored. If the monitoring detects a failure, then the cloud pipeline is adapted to the failure.