2018

Object-based reasoning in VQA

Desta, Mikyas T., Chen, Larry, Kornuta, Tomasz

Understand

Visual Question Answering (VQA) is a novel problem domain where multi-modal inputs must be processed in order to solve the task given in the form of a natural language.

As the solutions inherently require to combine visual and natural language processing with abstract reasoning, the problem is considered as AI-complete.
Recent advances indicate that using high-level, abstract facts extracted from the inputs might facilitate reasoning.
Following that direction we decided to develop a solution combining state-of-the-art object detection and reasoning modules.

Built on

Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from rgbd images
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus · 2012
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Original
T. Mikolov, K. Chen, G. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
D. Bahdanau, K. Cho, and Y. Bengio · 2014
Earlier work this paper cites.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Original
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Earlier work this paper cites.
Neural turing machines
Original
A. Graves, G. Wayne, and I. Danihelka · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Earlier work this paper cites.
Towards a visual turing challenge
Original
M. Malinowski and M. Fritz · 2014
Earlier work this paper cites.
GloVe: Global vectors for word representation
J. Pennington, R. Socher, and C. Manning · 2014
Earlier work this paper cites.
Memory networks
Original
J. Weston, S. Chopra, and A. Bordes · 2014
Earlier work this paper cites.
Deep compositional question answering with neural module networks. arxiv preprint
Original
J. Andreas, M. Rohrbach, T. Darrell, and D. Klein · 2015
Earlier work this paper cites.
VQA: Visual Question Answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Earlier work this paper cites.
Deep learning
Y. LeCun, Y. Bengio, and G. Hinton · 2015
Earlier work this paper cites.

Similar

Image question answering: A visual semantic embedding model and a new dataset
M. Ren, R. Kiros, and R. Zemel · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
Visual madlibs: Fill in the blank description generation and question answering
L. Yu, E. Park, A. C. Berg, and T. L. Berg · 2015
Cited alongside, same era.
Tensorflow: Large-scale machine learning on heterogeneous distributed systems
Original
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al · 2016
Cited alongside, same era.
Human attention in visual question answering: Do humans and deep networks look at the same regions?
Original
A. Das, H. Agrawal, C. L. Zitnick, D. Parikh, and D. Batra · 2016
Cited alongside, same era.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Original
A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach · 2016
Cited alongside, same era.
Region-based convolutional networks for accurate object detection and segmentation
R. Girshick, J. Donahue, T. Darrell, and J. Malik · 2016
Cited alongside, same era.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Original
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2016
Cited alongside, same era.
Adaptive computation time for recurrent neural networks
Original
A. Graves · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Speed/accuracy trade-offs for modern convolutional object detectors
Original
J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Cited alongside, same era.

Then

Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola · 2016
Later among the works it cites.
Bottom-up and top-down attention for image captioning and vqa
Original
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2017
Later among the works it cites.
Detecting visual relationships with deep relational networks
B. Dai, Y. Zhang, and D. Lin · 2017
Later among the works it cites.
Neural collaborative filtering
X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua · 2017
Later among the works it cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick · 2017
Later among the works it cites.
Inferring and executing programs for visual reasoning
Original
J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. L. Zitnick, and R. Girshick · 2017
Later among the works it cites.
One model to learn them all
Original
L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit · 2017
Later among the works it cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al · 2017
Later among the works it cites.
Ask your neurons: A deep learning approach to visual question answering
M. Malinowski, M. Rohrbach, and M. Fritz · 2017
Later among the works it cites.
Recurrent relational networks for complex relational reasoning
Original
R. B. Palm, U. Paquet, and O. Winther · 2017
Later among the works it cites.
A simple neural network module for relational reasoning
Original
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap · 2017
Later among the works it cites.
Visual question answering: A tutorial
D. Teney, Q. Wu, and A. van den Hengel · 2017
Later among the works it cites.
FVQA: Fact-based visual question answering
P. Wang, Q. Wu, C. Shen, A. Dick, and A. van den Hengel · 2017
Later among the works it cites.
Visual question answering: A survey of methods and datasets
Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2017
Later among the works it cites.
Weakly-supervised visual grounding of phrases with linguistic structures
F. Xiao, L. Sigal, and Y. J. Lee · 2017
Later among the works it cites.
Interpretable counting in visual question answering
A. Trott, C. Xiong, and R. Socher · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…