What value do explicit high level concepts have in vision to language problems?

Understand

Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

• This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text.
• We propose here a method of incorporating high-level concepts into the very successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art performance in both image captioning and visual question answering.
• We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance.