Modifying Cleaning Method in Big Data Analytics Process using Random Forest Classifier

摘要

Accurate data is a key success factor influencing the performance of data analytics results, especially for the detection and prediction purpose. Nowadays, Big Data analytics (BDA) is used to analyze the sheer volume of data available in an organization. These data quality must be maintained in order to obtain correct alert and valuable insights from the rapidly changing data of high volume, velocity, variety, veracity, and value. This paper aim is to modify existing framework of big data analytics by improving an important step in pre-processing (i.e. Data Cleaning). Initially, feature selection based on Random Forest is used to extract effective features. Then, two classifier algorithms (i.e. Random Forest classifier and Linear SVM classifier) are applied to train using the dataset to classify data quality and to develop an intelligent model. In evaluation, our experimental results show a consistent accuracy of Random Forest and Linear Regression around 90%. Using this approach, we expect to provide a set of cleaned data for further processing. Besides, analysts can benefit from this system in data analytical process in cleaning stage and conclude that the data is cleaned. Finally, a comparison is presented between available functions which are used to handle missing values with the developed system.