大规模网络异常流量实时云监测平台研究

摘要

Concealment of the network security problems appear increasingly strengthen, more durable, lethality spread more widely. A single or a few data sources is dififcult to ifnd more concealed abnormal network events. Meaning while, facing the huge-scale data some methods such as data mining, classiifcation, neural network, association rules, decision algorism, as the reason itself, are still existing the bottlenecks in the computing power. Base on the big data platform, the article puts forward a real-time monitoring system architecture to detect the abnormal trafifc in the massive network. The article discusses the key technologies and methods. The platform build up an architecture combining the oflfine batch computing and real-time streaming processing together. Through the analysis of the lfow rate, security logs and other large source data, it implements to monitor the network at instance and detect the abnormal lfow in real-time, such as DDoS attack, worms, scanning, and password probe.%网络安全问题呈现出隐蔽性越发增强、攻击更加持久、杀伤力波及更广等特征。单一或少数的数据源很难发现更加隐蔽的异常事件，同时一些针对入侵检测的数据挖掘、神经网络、关联规则、决策分类的算法由于算法本身的原因，对于大规模的数据存在计算能力上的瓶颈。文章提出了一种基于大数据平台的大规模网络异常流量实时监测系统架构，并讨论了关键技术和方法。该平台将离线的批处理计算和实时的流式处理计算相结合，通过对流量、日志等网络安全大数据的分析，实现对于DDoS、蠕虫、扫描、密码探测等异常流量的实时监测。