基于概率图模型的多模态数据联合检索算法

摘要

In order to improve the performance of the multi-modal retrieval,a semantic generation model of the multi-modal document and a joint retrieval algorithm for multi-modal data based on this model are proposed. The semantic generation model of the multi-modal document deems that the data of each modal in the document is generated by the same semantic concept, and the documents complies with the joint distribution of data of multiple modals. In order to simplify the solving process of model, the conditional probability of each modal can be calculated independently if assuming that the generation processes among all the modals′ data are mutual independent. During the multi-modal joint retrieval,the joint probability of the query data and the document under retrieval are calculated to obtain the similarity between them. The experimental results show that this algorithm has better retrieval performance than the two-step retrieval algorithm,semantic indexing algorithm and ranking learning algo-rithm. In addition,the algorithm can be applied to the document joint retrieval of data with three and more modals.%为了提高多模态检索的性能,提出一种多模态文档语义生成模型以及基于该模型的多模态数据联合检索算法.多模态文档语义生成模型认为文档中每个模态数据都是由相同的语义概念生成的,并且文档是多个模态数据的联合分布.为了简化模型的求解过程,假设各个模态数据之间的生成过程是相互独立的,于是可以对每个模态的条件概率进行单独计算.在多模态联合检索中,通过计算查询数据和待检索文档的联合概率来计算它们之间的相似度.实验结果表明,提出的方法与两步检索、语义索引和排序学习三种多模态检索方法相比具有更好的检索性能.此外,该方法可以扩展应用到具有三个及以上模态数据的文档联合检索中.