FastLogSim: A Quick Log Pattern Parser Scheme Based on Text Similarity

摘要

Logs completely record all system events which can be used to reveal network security issue and analyse user behaviour. Since logs are stored in the form of unstructured data and there is no universal log retention standard, they can hardly be analysed directly. Most of the existing log parsers focus on the parsing accuracy and ignore the time performance while parsing the large-amount of logs. Therefore, this paper proposes FastLogSim, a fast log parsing scheme based on text similarity. To simplify the parsing workload, we perform deduplication on the logs after removing the key variable values to obtain the template. Subsequently, the similarity is computed to merge the similar templates and then obtain the log pattern. FastLogSim not only reduces the number of templates that need to be parsed from tens of millions to dozens, but also greatly improves the speed of pattern extraction. We evaluated FastLogSim on four real public log datasets. The experimental results show that when the FastLogSim process tens thousands of logs, it performs almost the same time as the mainstream log parser. However, when the number of logs exceeds ten million, FastLogSim is three times faster than previous state-of-the-art parsers. Hence, FastLogSim is appropriative for large-scale log pattern mining.