The amount of digital data is growing at a staggering pace and mostly in the form of text. The growth of data, including multimedia (text, images, video, and audio) poses challenges in developing Multimedia Information Retrieval Systems (MIRS). Today, MIRS uses one or two media as a query input, for example, Google using text and image. There are comprehensive information needs in multimedia, including video and audio media as query input that can increase the amount and variety of information in retrieval result. Also, it is difficult to organize the relationship between the query input and the retrieval result in the same context or semantically related. This paper proposes a Unified Concept-based MIRS using deep learning with Ontology to tackle these problems. There are three main processes of this research; the first is Indexing Process which consist of collecting multimedia data, creating the multimedia dataset, extracting multimedia features, identifying and classifying objects and media format with Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), then storing object indexed as the concepts. The second is Query Processing which consists of inputting multimedia query or object to identifying and classifying object becomes a concept, and then the third is Retrieval and Rank Process of concept or object indexed and concept of query input. The ontology organizes the relationship between the concepts. Resource of this research uses the Cultural Heritage domain. The Unified Concept-based MIRS shows the capability of the system to extract features of four media (text, image, audio, and video) as concept representation to identify the multimedia object from query input and retrieve the object in four types of media at once. The retrieval results performance with deep learning increase about 30%-40% than support vector machine technique, and the usage of ontology increases the retrieval relevant results about 30 % is compared with MIRS without Ontology.
展开▼
