Language Models for Image Captioning: The Quirks and What Works

Understand

Two recent approaches have achieved state-of-the-art results in image captioning.

• The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence.
• The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence.
• In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input.