Parallelized Jaccard-based learning method and MapReduce implementation for mobile devices recognition from massive network data

摘要

The ability of accurate and scalable mobile device recognition is critically important for mobile network operators and ISPs to understand their customers' behaviours and enhance their user experience. In this paper, we propose a novel method for mobile device model recognition by using statistical information derived from large amounts of mobile network traffic data. Specifically, we create a Jaccard-based coefficient measure method to identify a proper keyword representing each mobile device model from massive unstructured textual HTTP access logs. To handle the large amount of traffic data generated from large mobile networks, this method is designed as a set of parallel algorithms, and is implemented through the MapReduce framework which is a distributed parallel programming model with proven low-cost and high-efficiency features. Evaluations using real data sets show that our method can accurately recognise mobile client models while meeting the scalability and producer-independency requirements of large mobile network operators. Results show that a 91.5% accuracy rate is achieved for recognising mobile client models from 2 billion records, which is dramatically higher than existing solutions.