Sampling Method in Traffic Logs Analyzing

摘要

In this paper, we aim to quantify the amount of degradation and bias that sampling introduces with respect to the non-sampled traffic data taking into account different sampling rates, different sampling policies, different sampling population, and different analysis tasks. First, we analyze the impact of sampling on summation task. Second, we apply sampling method to aggregation by a particular dimension task. We find that the relative error of different keys in aggregation are very different which will greatly limit the effect of sampling method on data compression when the application has strict limit to maximum relative error. So we implement a novel reservoir sampling policy based on our application and furthermore optimize it by combine static sampling policy with reservoir sampling policy. The results demonstrate that the proposed method can effectively control the maximum relative error while maintain data compression rate comparable to existing static sampling methods. Finally we analyze the user loss rate as a function of sampling step under system sampling policy. Through deeply inspect the number of logs for each user, we find the reason why large user loss rate occurs. Our results can provide useful reference for quick approximate analysis using sampling method in traffic logs analyzing area.