Reference Based on Adaptive Attention Mechanism for Image Captioning*

摘要

Image captioning, as a conjunction of computer vision and natural language processing, is receiving increasing attention in recent years. Most existing methods are using the attention-based CNN-RNN frameworks, which can understand images effectively and generate more natural descriptions. The recent research discovers that the expression of an event can effectively promote people's understanding and result in successful detection in objects and actions. Inspired by the textual information, in this paper, we propose a Reference based on Adaptive Attention Mechanism (R-AAM) model adding the reference sentence into the attention system to correct the area where the image is concentrated. Both in the training and testing processes, the reference sentence is selected by computing the largest consensus score among the nearest images in the training data set. The reference sentence associated with an image can help select the highlight region for the generating word in time sequence. The result has shown that the generated sentences utilizing the reference sentence express the richer semantic information and fix the wrong recognition phenomenon. Our proposed R-AAM method achieves comparable performances on the well-known public dataset MSCOCO with five popular evaluation metrics.