Attention Enhanced Single Stage Multimodal Reasoner

摘要

In this paper, we propose an Attention Enhanced Single Stage Multimodal Reasoner (ASSMR) to tackle the object referral task in the self-driving car scenario. We extract features from each modality and establish attention mechanisms to jointly process them. The Key Words Extractor (KWE) is used to extract the attribute and position/scale information of the target in the command, which are used to score the corresponding features through the Position/Scale Attention Module (P/SAM) and the Object Attention Module (OAM). Based on the attention mechanism, the effective part of the position/scale feature, the object attribute feature and the semantic feature of the command is enhanced. Finally, we map different features to a common embedding space to predict the final result. Our method is based on the simplified version of the Talk2Car dataset, and scored on 66.4 AP50 on the test set, while using the official region proposals.