Training SVM email classifiers using very large imbalanced dataset

摘要

The Internet has been flooded with spam emails, and during the last decade therenhas been an increasing demand for reliable anti-spam email filters. The problemnof filtering emails can be considered as a classification problem in the field ofnsupervised learning. Theoretically, many mature technologies, for example,nsupport vector machines (SVM), can be used to solve this problem. However, innreal enterprise applications, the training data are typically collected via honeypotsnand thus are always of huge amounts and highly biased towards spam emails.nThis challenges both efficiency and effectiveness of conventional technologies.nIn this article, we propose an undersampling method to compress and balance thentraining set used for the conventional SVM classifier with minimal informationnloss. The key observation is that we can make a trade-off between training set sizenand information loss by carefully defining a similarity measure between datansamples. Our experiments show that the SVM classifier provides a betternperformance by applying our compressing and balancing approach.