On-Board-Unit Data: A Big Data Platform for Scalable storage and Processing

摘要

This paper proposes and assesses a Big Data Platform for effective storage and analysis of On Board Unit (OBU) data related to the mobility of trucks in Belgium. The large volume and the streaming nature of the OBU data requires the setup of a big data platform for an efficient collection, storage and analysis. The solution relies on (i) the Hadoop Distributed File System (HDFS) to store data, (ii) the Apache Parquet format for data compression and columnar storage, and (iii) Spark for parallel and streaming processing of data. Data replication, compression and columnar storage ensure robustness to node failure, data distribution, and faster access to data.