2016

Measuring Machine Intelligence Through Visual Question Answering

Zitnick, C. Lawrence, Agrawal, Aishwarya, Antol, Stanislaw et al.

Understand

As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence.

A common approach is to propose tasks for which a human excels, but one which machines find difficult.
However, an ideal task should also be easy to evaluate and not be easily gameable.
We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence.

Built on

AI: The tumultuous history of the search for artificial intelligence
D. Crevier · 1993
Earlier work this paper cites.
Learning the semantics of words and pictures
K. Barnard and D. Forsyth · 2001
Earlier work this paper cites.
Vizwiz: nearly real-time answers to visual questions
J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, et al · 2010
Earlier work this paper cites.
Every picture tells a story: Generating sentences from images
A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth · 2010
Earlier work this paper cites.
Midge: Generating image descriptions from computer vision detections
M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III · 2012
Earlier work this paper cites.
Neil: Extracting visual knowledge from web data
X. Chen, A. Shrivastava, and A. Gupta · 2013
Earlier work this paper cites.
Paraphrase-driven learning for open question answering
A. Fader, L. S. Zettlemoyer, and O. Etzioni · 2013
Earlier work this paper cites.
Framing image description as a ranking task: Data, models and evaluation metrics
M. Hodosh, P. Young, and J. Hockenmaier · 2013
Earlier work this paper cites.
Babytalk: Understanding and generating simple image descriptions
G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg · 2013
Earlier work this paper cites.
Mctest: A challenge dataset for the open-domain machine comprehension of text
M. Richardson, C. J. Burges, and E. Renshaw · 2013
Earlier work this paper cites.
Bringing semantics into focus using visual abstraction
C. L. Zitnick and D. Parikh · 2013
Earlier work this paper cites.
Learning the visual interpretation of sentences
C. L. Zitnick, D. Parikh, and L. Vanderwende · 2013
Earlier work this paper cites.

Similar

Zero-shot learning via visual abstraction
S. Antol, C. L. Zitnick, and D. Parikh · 2014
Cited alongside, same era.
Learning everything about anything: Webly-supervised visual concept learning
S. K. Divvala, A. Farhadi, and C. Guestrin · 2014
Cited alongside, same era.
Comparing automatic evaluation measures for image description
D. Elliott and F. Keller · 2014
Cited alongside, same era.
Open question answering over curated and extracted knowledge bases
A. Fader, L. Zettlemoyer, and O. Etzioni · 2014
Cited alongside, same era.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Cited alongside, same era.
Researchers announce advance in image-recognition software
J. Markoff · 2014
Cited alongside, same era.
Joint video and text parsing for understanding events and answering queries
K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S.-C. Zhu · 2014
Cited alongside, same era.
Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick · 2015
Cited alongside, same era.
Mind’s eye: A recurrent visual representation for image caption generation
X. Chen and C. Lawrence Zitnick · 2015
Cited alongside, same era.
Exploring nearest neighbor approaches for image captioning
Original
J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick · 2015
Cited alongside, same era.

Then

Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Later among the works it cites.
From captions to visual concepts and back
H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, et al · 2015
Later among the works it cites.
Are you talking to a machine? dataset and methods for multilingual image question
H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu · 2015
Later among the works it cites.
Visual turing test for computer vision systems
D. Geman, S. Geman, N. Hallonquist, and L. Younes · 2015
Later among the works it cites.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Later among the works it cites.
Unifying visual-semantic embeddings with multimodal neural language models
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2015
Later among the works it cites.
Deep captioning with multimodal recurrent neural networks (m-rnn)
J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille · 2015
Later among the works it cites.
Cider: Consensus-based image description evaluation
R. Vedantam, C. Lawrence Zitnick, and D. Parikh · 2015
Later among the works it cites.
Learning common sense through visual abstraction
R. Vedantam, X. Lin, T. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Later among the works it cites.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Later among the works it cites.
Towards ai-complete question answering: A set of prerequisite toy tasks
Original
J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov · 2015
Later among the works it cites.
Adopting abstract images for semantic scene understanding
C. L. Zitnick, R. Vedantam, and D. Parikh · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…