An Analysis Platform of Road Traffic Management System Log Data Based on Distributed Storage and Parallel Computing Techniques

摘要

Road traffic management system generates a huge number of log data which can help the managers to monitor the drivers' behavior, identify traffic condition and extract some useful information. However, due to the high production rate and diversity of log data, it still is a big challenge to store and analyze them. Applying distributed storage and parallel computing in the clusters can improve the respond speed of searching and analyzing big data. In this work, we design a platform which integrates some distributed storage and parallel computing technologies i.e. Hadoop, Spark, Hive, Flume etc. to collect, store and analyze the mass log data. Moreover, the platform can timely display analysis results in graphical format through integrating some traditional technologies. The experimental results show that combining Flume and Hive is reliable and flexible in collecting and storing log data. Moreover, the query operation on mass log data by using Spark techniques spends less time than that in traditional ways, which verifies the success of our solution.