Research on mobile internet big data detecting method for the redundant data

摘要

To address the low efficiency of the traditional cleaning method, this paper presents a build path tree clean method based on split method for identification of the redundant data, through the traditional mobile internet big data cleaning process for identifying the redundant data is analysed, by using median filtering algorithm, the features of redundant data are extracted. Redundant data is classified by support vector machine (SVM), and the redundant data is identified by self-organising feature map. Based on this, the redundant data identification model is built, which can clean the redundant data in mobile internet big data. Comparing with the classical methods, the simulation results show that the proposed method has the advantages of high accuracy, good stability, high recall rate, short time consuming and low energy consumption.