A Distributed Intelligent Algorithm Applied to Imbalanced Data

摘要

Data mining means to find valuable information in database or data sets. For imbalanced data, there are extremely low number of samples in database or data sets and it is not easy to solve these problems by traditional methods of data mining. In this paper, a distributed intelligent algorithm is proposed to imbalanced data. Apache Spark is implemented as the distributed framework in the proposed distributed intelligent algorithm, and its cluster computing framework with in-memory data processing engine can do analytic on large volumes of data. In the distributed framework, Apache Spark with synthetic minority oversampling technique (SMOTE) is proposed to process imbalanced data first. Thereafter, the support vector machine (SVM) is used to classify imbalanced data. The zoo data set from UCI repository is used to verify the correctness of the proposed algorithm. The results of the proposed distributed intelligent algorithm can get better performance than these compared traditional classifiers.