Robust local outlier detection with statistical parameter for big data

摘要

With the rapid expansion of data scale, big data mining and analytics has attracted increasing attention. Outlier detection as an important task of data mining is widely used in many applications. However, conventional outlier detection methods begin to have difficulty handling large datasets. In addition, most existing outlier detection methods typically can only identify global outliers and are over sensitive to parameters variation. In this paper, we propose a novel method for robust local outlier detection with statistical parameter which incorporates the new ideas in dealing with big data. This method not only can effectively identify both global and local outliers but also is associated with only one statistical parameter. Furthermore, the sole parameter can be easily determined without relying on users' domain knowledge. Most importantly, the method is insensitive to parameter variation, which ensures its superiority in robustness. The properties of the method are investigated and the performance is experimentally verified using synthetic and publicly available datasets. The experiments demonstrate the accuracy and efficiency of the method in identifying local outliers. Moreover, the method is also proved more robust to parameter variation than the well-known local outlier detection method LOF and two other representative outlier detection methods, DB and DBSCAN. The results show that the proposed method has superiority in handling big data.