2016

Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

Wu, Qi, Shen, Chunhua, Hengel, Anton van den et al.

Understand

Much recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text.
In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering.
We further show that the same mechanism can be used to incorporate external knowledge, which is critically important for answering high level visual questions.

Built on

Z. Wu and M. Palmer, “Verbs semantics and lexical selection,” in Proc. Conf. Association for Computational Linguistics , 1994
1994
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proc. Conf. Association for Computational Linguistics , 2002
2002
Earlier work this paper cites.
C. Zhang, J. C. Platt, and P. A. Viola, “Multiple instance boosting for object detection,” in Proc. Advances in Neural Inf. Process. Syst. , 2005
2005
Earlier work this paper cites.
S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005
2005
Earlier work this paper cites.
J. Vogel and B. Schiele, “Semantic modeling of natural scenes for content-based image retrieval,” Int. J. Comput. Vision , vol. 72, no. 2, pp. 133–157, 2007
2007
Earlier work this paper cites.
S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, Dbpedia: A nucleus for a web of open data . Springer, 2007
2007
Earlier work this paper cites.
K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: a collaboratively created graph database for structuring human knowledge,” in Proc. ACM SIGMOD Int. Conf. Management of Data , 2008
2008
Earlier work this paper cites.
A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2009
2009
Earlier work this paper cites.
L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing, “Object bank: A high-level image representation for scene classification & semantic feature sparsification,” in Proc. Advances in Neural Inf. Process. Syst. , 2010, pp. 1378–1386
2010
Earlier work this paper cites.
A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in Proc. Eur. Conf. Comp. Vis. , 2010
2010
Earlier work this paper cites.
B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu, “I2t: Image parsing to text description,” Proc. IEEE , vol. 98, no. 8, pp. 1485–1508, 2010
2010
Earlier work this paper cites.
D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager et al. , “Building Watson: An overview of the DeepQA project,” AI magazine , vol. 31, no. 3, pp. 59–79, 2010
2010
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. Int. Conf. Artificial Intell. & Stat. , 2010, pp. 249–256
2010
Earlier work this paper cites.
Y. Jia, M. Salzmann, and T. Darrell, “Learning cross-modality similarity for multinomial data,” in Proc. IEEE Int. Conf. Comp. Vis. , 2011
2011
Earlier work this paper cites.
V. Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in Proc. Advances in Neural Inf. Process. Syst. , 2011
2011
Earlier work this paper cites.
S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi, “Composing simple image descriptions using web-scale n-grams,” in Proc. Conf. Computational Natural Language Learning , 2011
2011
Earlier work this paper cites.
Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos, “Corpus-guided sentence generation of natural images,” in Proc. Conf. Empirical Methods in Natural Language Processing , 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proc. Advances in Neural Inf. Process. Syst. , 2012
2012
Earlier work this paper cites.
Y. Su and F. Jurie, “Improving image classification using semantic attributes,” Int. J. Comput. Vision , vol. 100, no. 1, pp. 59–77, 2012
2012
Earlier work this paper cites.
L.-J. Li, H. Su, Y. Lim, and L. Fei-Fei, “Objects as attributes for scene classification,” in Trends and Topics in Computer Vision . Springer, 2012, pp. 57–69
2012
Earlier work this paper cites.
M. Hodosh, P. Young, and J. Hockenmaier, “Framing image description as a ranking task: Data, models and evaluation metrics,” J. Arti. Intell. Res. , pp. 853–899, 2013
2013
Earlier work this paper cites.
G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Babytalk: Understanding and generating simple image descriptions,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 35, no. 12, pp. 2891–2903, 2013
2013
Earlier work this paper cites.
M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele, “Translating video content to natural language descriptions,” in Proc. IEEE Int. Conf. Comp. Vis. , 2013
2013
Earlier work this paper cites.
J. Berant, A. Chou, R. Frostig, and P. Liang, “Semantic Parsing on Freebase from Question-Answer Pairs.” in Proc. Conf. Empirical Methods in Natural Language Processing , 2013, pp. 1533–1544
2013
Earlier work this paper cites.
K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in Proc. Conf. Empirical Methods in Natural Language Processing , 2014
2014
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proc. Advances in Neural Inf. Process. Syst. , 2014
2014
Earlier work this paper cites.

Similar

A. Karpathy, A. Joulin, and F. F. Li, “Deep fragment embeddings for bidirectional image sentence mapping,” in Proc. Advances in Neural Inf. Process. Syst. , 2014
2014
Cited alongside, same era.
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2014
2014
Cited alongside, same era.
Q. V. Le and T. Mikolov, “Distributed representations of sentences and documents,” in Proc. Int. Conf. Mach. Learn. , 2014
2014
Cited alongside, same era.
Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, “Improving image-sentence embeddings using large weakly annotated photo collections,” in Proc. Eur. Conf. Comp. Vis. , 2014
2014
Cited alongside, same era.
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, “Grounded compositional semantics for finding and describing images with sentences,” Proc. Conf. Association for Computational Linguistics , 2014
2014
Cited alongside, same era.
M. Malinowski and M. Fritz, “A multi-world approach to question answering about real-world scenes based on uncertain input,” in Proc. Advances in Neural Inf. Process. Syst. , 2014, pp. 1682–1690
2014
Cited alongside, same era.
K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S.-C. Zhu, “Joint video and text parsing for understanding events and answering queries,” MultiMedia, IEEE , vol. 21, no. 2, pp. 42–70, 2014
2014
Cited alongside, same era.
Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “HCP: A Flexible CNN Framework for Multi-label Image Classification,” IEEE Trans. Pattern Anal. Mach. Intell. , 2014
2014
Cited alongside, same era.
W. Zaremba and I. Sutskever, “Learning to execute,” arXiv:1410.4615 , 2014
Original
2014
Cited alongside, same era.
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” Proc. Conf. Association for Computational Linguistics , vol. 2, 2014
2014
Cited alongside, same era.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comp. Vis. , 2014
2014
Cited alongside, same era.
R. Kiros, R. Salakhutdinov, and R. S. Zemel, “Multimodal neural language models.” in Proc. Int. Conf. Mach. Learn. , vol. 14, 2014, pp. 595–603
2014
Cited alongside, same era.

Then

A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2015
2015
Later among the works it cites.
K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” in Proc. Int. Conf. Mach. Learn. , 2015
2015
Later among the works it cites.
X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars, “Guiding Long-Short Term Memory for Image Caption Generation,” in Proc. IEEE Int. Conf. Comp. Vis. , 2015
2015
Later among the works it cites.
D. Geman, S. Geman, N. Hallonquist, and L. Younes, “Visual Turing test for computer vision systems,” Proceedings of the National Academy of Sciences , vol. 112, no. 12, pp. 3618–3623, 2015
2015
Later among the works it cites.
H. Xu and K. Saenko, “Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering,” arXiv preprint arXiv:1511.05234 , 2015
Original
2015
Later among the works it cites.
K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and R. Nevatia, “ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering,” arXiv preprint arXiv:1511.05960 , 2015
Original
2015
Later among the works it cites.
A. Jiang, F. Wang, F. Porikli, and Y. Li, “Compositional Memory for Visual Question Answering,” arXiv preprint arXiv:1511.05676 , 2015
Original
2015
Later among the works it cites.
Y. Zhu, C. Zhang, C. Ré, and L. Fei-Fei, “Building a Large-scale Multimodal Knowledge Base for Visual Question Answering,” arXiv:1507.05670 , 2015
Original
2015
Later among the works it cites.
X. Lin and D. Parikh, “Don’t Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2015
2015
Later among the works it cites.
F. Sadeghi, S. K. Kumar Divvala, and A. Farhadi, “VisKE: Visual Knowledge Extraction and Question Answering by Visual Verification of Relation Phrases,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , June 2015
2015
Later among the works it cites.
J. Pont-Tuset, P. Arbeláez, J. Barron, F. Marques, and J. Malik, “Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation,” in IEEE Trans. Pattern Anal. Mach. Intell. , March 2015
2015
Later among the works it cites.
R. Girshick, “Fast r-cnn,” in Proc. IEEE Int. Conf. Comp. Vis. , 2015, pp. 1440–1448
2015
Later among the works it cites.
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Proc. Advances in Neural Inf. Process. Syst. , 2015, pp. 91–99
2015
Later among the works it cites.
R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based Image Description Evaluation,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2015
2015
Later among the works it cites.
J. Jin, K. Fu, R. Cui, F. Sha, and C. Zhang, “Aligning where to see and what to tell: image caption with region-based attention and scene factorization,” arXiv:1506.06272 , 2015
Original
2015
Later among the works it cites.
J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, “Exploring nearest neighbor approaches for image captioning,” arXiv preprint arXiv:1505.04467 , 2015
Original
2015
Later among the works it cites.
L. Ma, Z. Lu, and H. Li, “Learning to Answer Questions From Image using Convolutional Neural Network,” arXiv:1506.00333 , 2015
Original
2015
Later among the works it cites.
B. Zhou, Y. Tian, S. Sukhbaatar, A. Szlam, and R. Fergus, “Simple baseline for visual question answering,” arXiv preprint arXiv:1512.02167 , 2015
Original
2015
Later among the works it cites.
Q. Wu, C. Shen, A. v. d. Hengel, L. Liu, and A. Dick, “What Value Do Explicit High Level Concepts Have in Vision to Language Problems?” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
Q. Wu, P. Wang, C. Shen, A. Dick, and A. v. d. Hengel, “Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei, “Visual7W: Grounded Question Answering in Images,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Deep Compositional Question Answering with Neural Module Networks,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, “Stacked Attention Networks for Image Question Answering,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
H. Noh, P. H. Seo, and B. Han, “Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “Cnn-rnn: A unified framework for multi-label image classification,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2016
2016
Closest in time.
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Learning to compose neural networks for question answering,” in Proc. Conf. of North American Chapter of Association for Computational Linguistics , 2016
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…