Network Event Correlation Using Unsupervised Machine Learning Algorithms

摘要

We have successfully implemented a two-stage event correlation model for intrusion detection system (IDS) alerts. The model is designed to automate alert and incidents management and reduce the workload on an IDS analyst. We achieve this correlation by clustering similar alerts together, thus allowing the analyst to only look at a few clusters instead of hundreds or thousands of alerts. The first stage of this model uses an artificial neural network (ANN)- based autoassociator. The autoassociator is trained to reproduce each alert at its output, and it uses the error metric between its input and output to cluster similar alerts together. The accuracy of the system is improved by adding another machine-learning stage which attempts to combine closely related clusters produced by the first stage into super-clusters. The second stage uses the Expectation Maximisation (EM) clustering algorithm. The model and performance of this model are tested with intrusion alerts generated by a Snort IDS on DARPA's 1999 IDS evaluation data as well as incidents.org alerts.