An Electric Power Sensor Data Oriented Data Cleaning Solution

摘要

With the development of Smart Grid Technology, more and more electric power sensor data are utilized in various electric power systems. To guarantee the effectiveness of such systems, it is necessary to ensure the quality of electric power sensor data, especially when the scale of electric power sensor data is large. In the field of large-scale electric power sensor data cleaning, the computational efficiency and accuracy of data cleaning are two vital requirements. In order to satisfy these requirements, this paper presents an electric power sensor data oriented data cleaning solution, which is composed of a data cleaning framework and a data cleaning method. Based on Hadoop, the given framework is able to support large-scale electric power sensor data acquisition, storage and processing. Meanwhile, the proposed method which achieves outlier detection and reparation is implemented on the basis of a time-relevant k-means clustering algorithm in Spark. The feasibility and effectiveness of the proposed method is evaluated on a data set which originates from charging piles. Experimental results show that the proposed data cleaning method is able to improve the data quality of electric power sensor data by finding and repairing most outliers. For large-scale electric power sensor data, the proposed data cleaning method has high parallel performance and strong scalability.