Systematic approach to determine source of data quality issue in data flow in an enterprise

摘要

A method may include applying periodically a data validation rule to data transformed through a data processing system, wherein the data validation rule applies aspects selected from a group consisting of data value range, specific data values, and relationship with other data entries; responsive to detecting a violation of the data validation rule, identifying a portion of the transformed data for lineage assessment; examining the identified transformed data iteratively upstream at a previous transformation node in a lineage graph, until the method detects a node where the violation of the data validation rule can't be reproduced; creating a separate node in a distributed network for each of the previous transformation nodes in the lineage graph; and identifying the separate node in the distributed network introducing the violation of the data validation rule.