Filtering error log as time series in complex service-based storage systems

摘要

Mining log pattern to analyze the faults in large scale distributed system is affected by the existence of redundant and ambiguous noisy error logs. While existing works try to compress logs in a coarse granularity from temporal and spatial view to remove the redundancy, they fail to reserve those ambiguous logs that might truly relate to a fault, which misleads the fault characterizing result. By modeling error logs as time series and examining the similarity between trash error log template and target error log, the ambiguous error logs are kept and the affected patterns can be effectively removed. Experiments in a practical complex service-based storage show that up to 92% of the affected patterns can be filtered.