A High-Speed Network Data Acquisition System Based on Big Data Platform

摘要

This paper designs a high-speed network data acquisition system based on big data platform with good expansibility and high real-time, aiming at the need of enterprises or organizations to acquire high-speed network data efficiently in network intrusion detection. Taking into account the storage capacity and computing power requirements of high-speed network traffic capture and processing, the whole system is divided into two parts. The front-end adopts high-performance network traffic distributor based on hardware, and uses it to divide the high-speed network traffic into multiple low-speed branch traffic. The back-end data acquisition and processing system is implemented by clustering of normal PC, and the software system is optimized by introducing big data technology. The use of distributed storage and computing improve data processing performance. The system acquires the complete network data through distribution, capture, cache, layered protocol analysis, session reassembly, application layer data extraction and so on, which can provide reference for intrusion detection.