An Anomaly Data Mining Method for Mass Sensor Networks Using Improved PSO Algorithm Based on Spark Parallel Framework

摘要

Accurate detection and capture of anomaly data in complex network data stream is an important part of ensuring network security. Traditional methods cannot adapt to the high dynamic changes of abnormal data characteristics in complex network. Thus, the detection accuracy is reduced. In this paper, a k-means parallel clustering algorithm is proposed. It is optimized by particle swarm optimization with dynamic adaptive inertia weight (dsPSOK-means). And it is used to mine the anomaly data for mass sensor networks. The inertia weight is dynamically adjusted through the fitness function, so that the dsPSO algorithm has the adaptive characteristics. Then, the output of the dsPSO algorithm is taken as the input of the k-means algorithm. Thus, the intelligence and self-adaptability of the k-means algorithm in selecting the initial center point is improved. Finally, with the help of Spark platform, the parallelization of dsPSOK-means clustering algorithm in the clustering environment is designed and implemented. It is shown by the experimental results that the traffic among nodes in the execution process can be effectively reduced by the dsPSOK-means algorithm. And the accuracy of abnormal data mining in complex network data flow is 5% higher than that of the comparison algorithm on average.