What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?

摘要

In neural image captioning systems, a recurrent neural network (RNN) istypically viewed as the primary `generation' component. This view suggests thatthe image features should be `injected' into the RNN. This is in fact thedominant view in the literature. Alternatively, the RNN can instead be viewedas only encoding the previously generated words. This view suggests that theRNN should only be used to encode linguistic features and that only the finalrepresentation should be `merged' with the image features at a later stage.This paper compares these two architectures. We find that, in general, latemerging outperforms injection, suggesting that RNNs are better viewed asencoders, rather than generators.