2015

Yin and Yang: Balancing and Answering Binary Visual Questions

Zhang, Peng, Goyal, Yash, Summers-Stay, Douglas et al.

Understand

The complex compositional structure of language makes problems at the intersection of vision and language challenging.

But language also provides a strong prior that can result in good superficial performance, without the underlying models truly understanding the visual content.
This can hinder progress in pushing state of art in the computer vision aspects of multi-modal AI.
In this paper, we address binary Visual Question Answering (VQA) on abstract scenes.

Built on

Best practices for convolutional neural networks applied to visual document analysis
P. Y. Simard, D. Steinkraus, and J. C. Platt · 2003
Earlier work this paper cites.
Hunpos - an open source trigram tagger
P. Halácsy, A. Kornai, and C. Oravecz · 2007
Earlier work this paper cites.
Using annotator rationales to improve machine learning for text categorization
O. Zaidan, J. Eisner, and C. Piatko · 2007
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
VizWiz: Nearly Real-time Answers to Visual Questions
J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh · 2010
Earlier work this paper cites.
Annotator rationales for visual recognition
J. Donahue and K. Grauman · 2011
Earlier work this paper cites.
Identifying relations for open information extraction
A. Fader, S. Soderland, and O. Etzioni · 2011
Earlier work this paper cites.
Unbiased look at the bias
A. Torralba and A. Efros · 2011
Earlier work this paper cites.
Attributes for classifier feedback
A. Parkash and D. Parikh · 2012
Earlier work this paper cites.
Simultaneous active learning of classifiers & attributes via relative feedback
A. Biswas and D. Parikh · 2013
Earlier work this paper cites.
Some improvements on deep convolutional neural network based image classification
Original
A. G. Howard · 2013
Earlier work this paper cites.
Distributed Representations of Words and Phrases and their Compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Bringing Semantics Into Focus Using Visual Abstraction
C. L. Zitnick and D. Parikh · 2013
Earlier work this paper cites.
Learning the Visual Interpretation of Sentences
C. L. Zitnick, D. Parikh, and L. Vanderwende · 2013
Earlier work this paper cites.

Similar

Zero-Shot Learning via Visual Abstraction
S. Antol, C. L. Zitnick, and D. Parikh · 2014
Cited alongside, same era.
Predicting object dynamics in scenes
D. F. Fouhey and C. Zitnick · 2014
Cited alongside, same era.
A Visual Turing Test for Computer Vision Systems
D. Geman, S. Geman, N. Hallonquist, and L. Younes · 2014
Cited alongside, same era.
Identifying relations for open information extraction
A. Karpathy, A. Joulin, and F.-F. Li · 2014
Cited alongside, same era.
Microsoft COCO: Common Objects in Context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
M. Malinowski and M. Fritz · 2014
Cited alongside, same era.
Explain Images with Multimodal Recurrent Neural Networks
Original
J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille · 2014
Cited alongside, same era.
Joint Video and Text Parsing for Understanding Events and Answering Queries
K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S. C. Zhu · 2014
Cited alongside, same era.
https://github.com/fchollet/keras.git , 2015
Keras: Theano-based deep learning library · 2015
Cited alongside, same era.
Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation
X. Chen and C. L. Zitnick · 2015
Cited alongside, same era.
Exploring nearest neighbor approaches for image captioning
Original
J. Devlin, S. Gupta, R. B. Girshick, M. Mitchell, and C. L. Zitnick · 2015
Cited alongside, same era.

Then

From Captions to Visual Concepts and Back
H. Fang, S. Gupta, F. N. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig · 2015
Closest in time.
Are you talking to a machine? dataset and methods for multilingual image question answering
H. Gao, J. Mao, J. Zhou, Z. Huang, and A. Yuille · 2015
Closest in time.
Learning to interpret and describe abstract scenes
L. Gilberto Mateos Ortiz, C. Wolff, and M. Lapata · 2015
Closest in time.
Deep Visual-Semantic Alignments for Generating Image Descriptions
A. Karpathy and L. Fei-Fei · 2015
Closest in time.
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2015
Closest in time.
Don’t just listen, use your imagination: Leveraging visual common sense for non-visual tasks
X. Lin and D. Parikh · 2015
Closest in time.
Ask your neurons: A neural-based approach to answering questions about images
M. Malinowski, M. Rohrbach, and M. Fritz · 2015
Closest in time.
Exploring models and data for image question answering
M. Ren, R. Kiros, and R. Zemel · 2015
Closest in time.
Viske: Visual knowledge extraction and question answering by visual verification of relation phrases
F. Sadeghi, S. K. Kumar Divvala, and A. Farhadi · 2015
Closest in time.
Learning common sense through visual abstraction
R. Vedantam, X. Lin, T. Batra, C. L. Zitnick, and D. Parikh · 2015
Closest in time.
Show and Tell: A Neural Image Caption Generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Closest in time.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio · 2015
Closest in time.
Adopting Abstract Images for Semantic Scene Understanding
C. L. Zitnick, R. Vedantam, and D. Parikh · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…