Advanced Streaming Data Cleansing

摘要

A frequent problem experienced throughout industry is that of missing or poor quality data in data historians. This can have various causes, such as field instrument failures, loss of communication, or even issues with the setup of the historian itself. The end result is that data required to perform analyses needed to improve facility operations may be unavailable. This generally incurs delays, as the data analyst must manually “clean up” the data before using it, or could even result in erroneous conclusions if the data is used as is without any corrections. In this paper, a novel multivariate statistical method is proposed to detect incorrect data values and reconstruct corrected values to be stored in the historian. This method works on streaming data, and thus makes its corrections continuously in near real-time. The method has been successfully tested in a laboratory setting using real operating data from a Chevron facility. Chevron plans to test the data error detection and reconstruction method in the field in the near future. Use of this method will ensure that good quality data for needed analyses is available in the data historian, and will save analyst time as well.