Multimedia retrieval is a search and ranking task defined over multiple modalities. These modalities include speech, image, and text, which provide different views of the multimedia object. Queries to a multimedia retrieval system often take the form of a text only query and return a ranked result set which combines these multiple views. The text only query includes multiple phrases which identify features of a specific view. This multiview problem presents a challenge in mapping these phrases into the correct view feature space. A second challenge for the multimedia retrieval system is in building a ranking model which considers the unique feature space of each view. In this paper, we propose a hierarchical multimedia multiview rank learning model, called MultiRank, to overcome the challenges of this unique ranking problem. The first layer of our model uses natural language processing techniques to identify view specific phrases and output a ranked mapping of the phrases into their respective views. Next, we model the individual feature space for each multimedia view and create a view specific model using gradient boosted regression trees. The ranked set from each unique view is then passed to the final layer of the hierarchy, where the model generates a final ranked result set. We show that our multiview rank learning approach is effective by evaluating the methods using a large Internet video repository, queries, and ground truth, from the TRECVid evaluations.
展开▼
