Semantic Understanding for Video Retrieval with Temporal Multimodal Fusion Analysis

摘要

As the use of video increases in the digital library, offering effective retrieval service for video is a great demand. However, compared with text and images, video lacks of obvious semantics although video has rich multimodal information, such as text transcripts, visual/audio features and temporal structure. Therefore, understanding semantics embedded in video is necessary for video retrieval. In this paper, we propose a comprehensive approach to semantic understanding of video through automatic annotation with temporal multimodal fusion analysis. Various media aspects are investigated, including meaningful words and contextual distribution in the transcript, visual/audio features, and most importantly, the temporal interval relations involved in video. TFIDF retrieval method with score propagation is used to discover the association between a shot and its corresponding transcript. Experiments on the TRECVID 2003 dataset show that our approach achieves high performance.