Hybrid Consensus Pruning of Ensemble Classifiers for Big Data Malware Detection

摘要

One of the major challenges for safeguarding the security of big data in the cloud is how to detect and prevent malicious software (malware). Despite of the fact that security and privacy are critical issues in big data, more research needs to be done in this area. As malware can affect the reliability of the data and subsequently the reputation of the system, it is critical to detect and remove malware from a system as early as possible. Recently, ensembles that combine a set of classifiers have been proposed as an efficient approach for malware detection. Unfortunately, the size, memory and processing requirements as well as the high cost of data transfer during training and operation make large ensemble classifiers unsuitable for big data in the cloud. To address this problem, we propose a new advanced ensemble pruning method, Hybrid Consensus Pruning (HCP), which is the first pruning algorithm that employs a fast consensus function to combine several classifier classes into one scheme. To test the effectiveness of the HCP method, we conducted experiments comparing its performance with Ensemble Pruning via Individual Contribution ordering (EPIC), Directed Hill Climbing Ensemble Pruning (DHCEP) and K-Means Pruning approaches for pruning very large ensemble classifiers for malware detection. The results of the experiments show that HCP achieved better results by producing better ensemble classifiers as compared to those created by EPIC, DHCEP and K-Means Pruning.