2015

Visual7W: Grounded Question Answering in Images

Zhu, Yuke, Groth, Oliver, Bernstein, Michael et al.

Understand

We have seen great progress in basic perceptual tasks such as object recognition and detection.

However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning.
Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding.
Previous works have established a loose, global association between QA sentences and images.

Built on

Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Matching words and pictures
K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan · 2003
Earlier work this paper cites.
Learning to detect unseen object classes by between-class attribute transfer
C. H. Lampert, H. Nickisch, and S. Harmeling · 2009
Earlier work this paper cites.
Vizwiz: nearly real-time answers to visual questions
J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, et al · 2010
Earlier work this paper cites.
Building Watson: An overview of the DeepQA project
D. Ferrucci et al · 2010
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Dating historical color images
F. Palermo, J. Hays, and A. A. Efros · 2012
Earlier work this paper cites.
Sun attribute database: Discovering, annotating, and recognizing scene attributes
G. Patterson and J. Hays · 2012
Earlier work this paper cites.
Political journalism: New challenges, new practices
R. Kuhn and E. Neveu · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Translating video content to natural language descriptions
M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, and B. Schiele · 2013
Earlier work this paper cites.
Learning the visual interpretation of sentences
C. L. Zitnick, D. Parikh, and L. Vanderwende · 2013
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
R. Girshick, J. Donahue, T. Darrell, and J. Malik · 2014
Earlier work this paper cites.
A neural network for factoid question answering over paragraphs
M. Iyyer, J. Boyd-Graber, L. Claudino, R. Socher, and H. Daumé III · 2014
Earlier work this paper cites.
Deep fragment embeddings for bidirectional image sentence mapping
A. Karpathy, A. Joulin, and L. Fei-Fei · 2014
Earlier work this paper cites.
Large-scale video classification with convolutional neural networks
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei · 2014
Earlier work this paper cites.
Referit game: Referring to objects in photographs of natural scenes
S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Earlier work this paper cites.

Similar

What are you talking about? text-to-image coreference
C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler · 2014
Cited alongside, same era.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Cited alongside, same era.
Seeing the arrow of time
L. C. Pickup, Z. Pan, D. Wei, Y. Shih, C. Zhang, A. Zisserman, B. Scholkopf, and W. T. Freeman · 2014
Cited alongside, same era.
Inferring the why in images
Original
H. Pirsiavash, C. Vondrick, and A. Torralba · 2014
Cited alongside, same era.
Linking people with ”their” names using coreference resolution
V. Ramanathan, A. Joulin, P. Liang, and L. Fei-Fei · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2014
Cited alongside, same era.
Grounded compositional semantics for finding and describing images with sentences
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. V. Le · 2014
Cited alongside, same era.
Deepface: Closing the gap to human-level performance in face verification
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf · 2014
Cited alongside, same era.
Deeppose: Human pose estimation via deep neural networks
A. Toshev and C. Szegedy · 2014
Cited alongside, same era.
Joint video and text parsing for understanding events and answering queries
K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S.-C. Zhu · 2014
Cited alongside, same era.

Then

Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Closest in time.
Are you talking to a machine? dataset and methods for multilingual image question answering
H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu · 2015
Closest in time.
Visual turing test for computer vision systems
D. Geman, S. Geman, N. Hallonquist, and L. Younes · 2015
Closest in time.
Draw: A recurrent neural network for image generation
Original
K. Gregor, I. Danihelka, A. Graves, and D. Wierstra · 2015
Closest in time.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Closest in time.
Where to buy it: Matching street clothing photos in online shops
M. H. Kiapour, X. Han, S. Lazebnik, A. C. Berg, and T. L. Berg · 2015
Closest in time.
Learning deep representations for ground-to-aerial geolocalization
T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays · 2015
Closest in time.
Learning to answer questions from image using convolutional neural network
Original
L. Ma, Z. Lu, and H. Li · 2015
Closest in time.
Ask your neurons: A neural-based approach to answering questions about images
M. Malinowski, M. Rohrbach, and M. Fritz · 2015
Closest in time.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
B. Plummer, L. Wang, C. Cervantes, J. Caicedo, J. Hockenmaier, and S. Lazebnik · 2015
Closest in time.
Exploring models and data for image question answering
M. Ren, R. Kiros, and R. Zemel · 2015
Closest in time.
ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Closest in time.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Closest in time.
Towards ai-complete question answering: a set of prerequisite toy tasks
Original
J. Weston, A. Bordes, S. Chopra, and T. Mikolov · 2015
Closest in time.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio · 2015
Closest in time.
Visual Madlibs: Fill in the blank Image Generation and Question Answering
L. Yu, E. Park, A. C. Berg, and T. L. Berg · 2015
Closest in time.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…