2017

Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge

Teney, Damien, Anderson, Peter, He, Xiaodong et al.

Understand

This paper presents a state-of-the-art model for visual question answering (VQA), which won the first place in the 2017 VQA Challenge.

VQA is a task of significant importance for research in artificial intelligence, given its multimodal nature, clear evaluation protocol, and potential real-world applications.
The performance of deep neural networks for VQA is very dependent on choices of architectures and hyperparameters.
To help further research in the area, we describe in detail our high-performing, though relatively simple model.

Built on

Zipfs law and the internet
L. A. Adamic and B. A. Huberman · 2002
Earlier work this paper cites.
ADADELTA: an adaptive learning rate method
Original
M. D. Zeiler · 2012
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Glove: Global Vectors for Word Representation
J. Pennington, R. Socher, and C. Manning · 2014
Earlier work this paper cites.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2014
Earlier work this paper cites.
VQA: Visual Question Answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
Original
K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and R. Nevatia · 2015
Earlier work this paper cites.
From captions to visual concepts and back
H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt, et al · 2015
Earlier work this paper cites.
Compositional Memory for Visual Question Answering
Original
A. Jiang, F. Wang, F. Porikli, and Y. Li · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Earlier work this paper cites.
Highway networks
Original
R. K. Srivastava, K. Greff, and J. Schmidhuber · 2015
Earlier work this paper cites.
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Original
H. Xu and K. Saenko · 2015
Earlier work this paper cites.

Similar

Learning to compose neural networks for question answering
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein · 2016
Cited alongside, same era.
Neural Module Networks
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein · 2016
Cited alongside, same era.
Language modeling with gated convolutional networks
Original
Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier · 2016
Cited alongside, same era.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Original
A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach · 2016
Cited alongside, same era.
Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering
Original
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Identity mappings in deep residual networks
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Revisiting visual question answering baselines
A. Jabri, A. Joulin, and L. van der Maaten · 2016
Cited alongside, same era.
CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning
Original
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick · 2016
Cited alongside, same era.
TrimZero: A Torch Recurrent Module for Efficient Natural Language Processing
J.-H. Kim, J. Kim, J.-W. Ha, and B.-T. Zhang · 2016
Cited alongside, same era.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
Original
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Cited alongside, same era.

Then

Graph-structured representations for visual question answering
D. Teney, L. Liu, and A. van den Hengel · 2016
Later among the works it cites.
Zero-shot visual question answering
D. Teney and A. van den Hengel · 2016
Later among the works it cites.
Stacked Attention Networks for Image Question Answering
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola · 2016
Later among the works it cites.
Yin and yang: Balancing and answering binary visual questions
P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh · 2016
Later among the works it cites.
Visual7W: Grounded Question Answering in Images
Original
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Bottom-up and top-down attention for image captioning and vqa
Original
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2017
Closest in time.
MUTAN: multimodal tucker fusion for visual question answering
Original
H. Ben-younes, R. Cadène, M. Cord, and N. Thome · 2017
Closest in time.
Visual Dialog
A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra · 2017
Closest in time.
Learning to reason: End-to-end module networks for visual question answering
Original
R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko · 2017
Closest in time.
Inferring and executing programs for visual reasoning
Original
J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, F. Li, C. L. Zitnick, and R. B. Girshick · 2017
Closest in time.
Show, ask, attend, and answer: A strong baseline for visual question answering
Original
V. Kazemi and A. Elqursh · 2017
Closest in time.
Visual question answering: A survey of methods and datasets
Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…