Cross-modal retrieval in challenging scenarios using attributes

摘要

Cross-modal retrieval is an important field of research today because of the abundance of multi-media data. In this work, we attempt to address two challenging scenarios that we may encounter in real-life cross-modal retrieval, but which are relatively unexplored in literature. First, due to the ever-increasing number of new categories of data, cross-modal algorithms should be able to generalize to categories which it has not seen during training. Second, the data that is available during testing may be degraded (for example, it has low resolution or noise) as compared to those available during training. Here, we evaluate how these adverse conditions affect the performance of the state-of-the-art cross-modal approaches. We also propose a unified framework that can handle all these diverse and challenging scenarios without any modification. In the proposed approach, the data from different modalities are projected into a common semantic preserving latent space in which semantic relations as given by the classname embeddings (attributes) are preserved. Extensive experiments on diverse cross-modal data including image-text, RGB-depth and comparison with the state-of-the-art approaches show the usefulness of the proposed approach for these challenging scenarios. (C) 2019 Elsevier B.V. All rights reserved.