OPTIMIZATION OF DATA PROCESSING JOB EXECUTION USING HASH TREES

摘要

Optimization of job execution includes generating a hash tree for an initial dataset, executing a data processing job against the initial dataset, the job including a set of calculations against data of the records of the initial dataset, generating, based on a modification to the initial dataset to produce a modified dataset, a hash tree for the modified dataset, comparing the hash tree for the initial dataset to the hash tree for the modified dataset, and identifying one or more blocks of the modified dataset that are updated compared to the initial dataset, and re-executing the job against the modified dataset. The re-executing performs only those calculations that use data of the records of the one or more blocks that are updated compared to the initial dataset, and does not perform other calculations that do not use data of the records of the one or more blocks.