Construction and Application of Ship Data Mining Platform Based on Spark

摘要

With the explosive growth of ship information data brought by the intellectualization and digitization of waterborne traffic, it has become more and more important to process and apply massive ship data quickly and accurately. In view of the growing maturity and wide application of Spark large data technology in data mining field, this paper, against the characteristics of ship AIS data, chooses a series of large data technologies based on Spark to build the AIS data mining platform. According to the basic process of mining and processing, the overall framework of the platform is divided into three modules: database, Spark computing and mining, visualization. The AIS data of Sunda Strait with high ship density is used to process and mine the actual data to verify the availability of the platform. The results show that the platform works well and the three modules can work normally. And the AIS data of the Strait are processed and mined quickly and accurately. According to different requirements, it can display bar charts, polyline charts, histogram, trajectory charts and thermal charts and other commonly used data analysis graphics to achieve the effect of navigation assistance.