Optimizing data processing across server clusters and data centers using checkpoint-based data replication

摘要

Aspects of the disclosure relate to optimizing data processing across server clusters and data centers using checkpoint-based data replication. A computing platform may determine to initiate a data processing job associated with identifying one or more features of a source dataset, and the data processing job may include multiple processing steps. Based on determining to initiate the data processing job, the computing platform may generate one or more commands directing one or more cluster server nodes associated with a data center to execute the multiple processing steps. The one or more commands may direct the one or more cluster server nodes to update a checkpoint table as each processing step is completed, and may further direct the one or more cluster server nodes to replicate processing results data to at least one other data center. Subsequently, the computing platform may send the generated commands to the cluster server nodes.