基于Spark的大规模网络流量分类研究

摘要

Internet traffic classification using machine learning has become a hot research topic in the field of network security. In order to improve the classification efficiency of large scale network flow, this paper introduces a parallel SVM algorithm to identify the network traffic, and proposes a real-time classification scheme for large scale network flow based on Spark. This method builds a classification model using parallel SVM algorithm, and then it is integrated with the latest flow processing framework for real-time classification of large-scale networks. Experimental results show that parallel SVM algorithm can greatly improve the training and classification speed of the network flow model, on the premise of little loss of precision.%机器学习算法处理流量分类问题已经成为网络安全领域一个研究热点。为了提高大规模网络流的分类效率，引入并行SVM算法来识别网络流量，提出了一种基于Spark平台的大规模网络流在线分类方案。该方案利用置信域牛顿法(TRON)并行优化线性SVM算法构建流量分类模型，然后融合最新的实时计算框架，实现对大规模网络流的在线识别。实验结果表明，利用并行SVM算法在损失较小精度的前提下可以加快网络流的模型训练和分类速度，符合大规模网络流在线分类的需要。