Common Semantic Representation Method Based on Object Attention and Adversarial Learning for Cross-Modal Data in IoV

摘要

With the significant development of the Internet of Vehicles (IoV), various modal data, such as image and text, are emerging, which provide data support for good vehicle networking services. In order to make full use of the cross-modal data, we need to establish a common semantic representation to achieve effective measurement and comparison of different modal data. However, due to the heterogeneous distributions of cross-modal data, there exists a semantic gap between them. Although some deep neural network (DNN) based methods have been proposed to deal with this problem, there still exist several challenges: the qualities of the modality-specific features, the structure of the DNN, and the components of the loss function. In this paper, for representing cross-modal data in IoV, we propose a common semantic representation method based on object attention and adversarial learning (OAAL). To acquire high-quality modality-specific feature, in OAAL, we design an object attention mechanism, which links the cross-modal features effectively. To further alleviate the heterogeneous semantic gap, we construct a cross-modal generative adversarial network, which contains two parts: a generative model and a discriminative model. Besides, we also design a comprehensive loss function for the generative model to produce high-quality features. With a minimax game between the two models, we can construct a shared semantic space and generate the unified representations for cross-modal data. Finally, we apply our OAAL on retrieval task, and the results of the experiments have verified its effectiveness.