Detecting loss of data received at a data processing pipeline of an online system during a specified time interval

摘要

An online system receives data and processes the data in a data processing pipeline. To data loss in the data processing pipeline, the online system determines a time interval during which each item of data is received and associates a set of counters with each time interval. For each time interval, an input counter is incremented for each data item received during the time interval and an output counter is incremented for each data item received during the time interval that was processed by the data processing pipeline. The online system compares an input number from the input counter and an output number from the output counter for each time interval. Based on a difference between the input number and output number for a time interval, the online system determines if a loss of data received during the time interval occurred. Lost Data are identified and processed.