Image captioning is a cross-domain task which connects computer vision,natural language processing and machine learning.As a key technology of multimodal processing,it has made remarkable progress in the recent years.Research on image caption generation has typically focused on generating a caption in English for an image, but generating Chinese caption is lack of research.In this paper,we propose a method generating Chinese image caption based on multimodal neural network.This method belongs to the family of encoder-decoder.Encoder based on convolutional neural network,consists of single-label visual feature extraction network and multi-label keyword prediction network.Decoder based on long short-term memory,consists of multimodal caption generation network. During the process of decoding,we propose four multimodal caption generation methods:CNIC-X,CNIC-H,CNIC-C and CNIC-HC.Experimental results on Chinese multimodal dataset Flickr8k-CN show that the proposed method outperforms state-of-the-art Chinese image captioning methods.%图像的自然语言描述(image captioning)是一个融合计算机视觉、自然语言处理和机器学习的跨领域课题.它作为多模态处理的关键技术,近年来取得了显著成果.当前研究大多针对图像生成英文摘要,而对于中文摘要的生成方法研究较少.该文提出了一种基于多模态神经网络的图像中文摘要生成方法.该方法由编码器和解码器组成,编码器基于卷积神经网络,包括单标签视觉特征提取网络和多标签关键词特征预测网络,解码器基于长短时记忆网络,由多模态摘要生成网络构成.在解码过程中,该文针对长短时记忆网络的特点提出了四种多模态摘要生成方法CNIC-X、CNIC-H、CNIC-C和CNIC-HC.在中文摘要数据集Flickr8k-CN上实验,结果表明该文提出的方法优于现有的中文摘要生成模型.
展开▼
