Re-ranking image-text matching by adaptive metric fusion

摘要

Image-text matching has drawn much attention recently with the rapid growth of multi-modal data. Many effective approaches have been proposed to solve this challenging problem, but limited effort has been devoted to re-ranking methods. Compared with the uni-modal re-ranking methods, modality heterogeneity is the major difficulty when designing a re-ranking method in the cross-modal field, which mainly lies in two aspects of different visual and textual feature spaces and different distributions in inverse directions. In this paper, we propose a heuristic re-ranking method called Adaptive Metric Fusion (AMF) for image-text matching. The method can obtain a better metric by adaptively fusing metrics based on two modules: 1) Cross-modal Reciprocal Encoding, which considers ranks in inverse directions to comprehensively evaluate a metric. The sentence retrieval and image retrieval have different distribution characteristics and galleries in different modalities, thus it is necessary to exploit them simultaneously for appropriate metric fusion. 2) Query Replacement Gap, which quantifies the gap between cross-modal and uni-modal similarities to alleviate the influence of different visual and textual feature spaces on the fused metric. The proposed re-ranking method can be implemented in an unsupervised way without requiring any human interaction or annotated data, and can be easily applied to any initial ranking result. Extensive experiments and analysis validate the effectiveness of our method on the large-scale MS-COCO and Flickr30K datasets. (C) 2020 Elsevier Ltd. All rights reserved.