An Efficient Network Log Anomaly Detection System using Random Projection Dimensionality Reduction

摘要

Network traffic is increasing all the time andnetwork services are becoming more complex and vulnerable.To protect these networks, intrusion detection systems are used.Signature-based intrusion detection cannot find previously unknownattacks, which is why anomaly detection is needed.However, many new systems are slow and complicated. Wepropose a log anomaly detection framework which aims tofacilitate quick anomaly detection and also provide visualizationsof the network traffic structure. The system preprocesses networklogs into a numerical data matrix, reduces the dimensionalityof this matrix using random projection and uses Mahalanobisdistance to find outliers and calculate an anomaly score foreach data point. Log lines that are too different are flagged asanomalies. The system is tested with real-world network data, andactual intrusion attempts are found. In addition, visualizations arecreated to represent the structure of the network data. We alsoperform computational time evaluation to ensure the performanceis feasible. The system is fast, finds real intrusion attempts anddoes not need clean training data.