2018

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Yi, Kexin, Wu, Jiajun, Gan, Chuang et al.

Understand

We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning.

Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question.
It then executes the program on the scene representation to obtain an answer.
Incorporating symbolic structure as prior knowledge offers three unique advantages.

Built on

Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Vision as bayesian inference: analysis by synthesis?
Alan Yuille and Daniel Kersten · 2006
Earlier work this paper cites.
Semantic parsing on freebase from question-answer pairs
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang · 2013
Earlier work this paper cites.
Learning dependency-based compositional semantics
Percy Liang, Michael I Jordan, and Dan Klein · 2013
Earlier work this paper cites.
A multi-world approach to question answering about real-world scenes based on uncertain input
M Malinowski and M Fritz · 2014
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Earlier work this paper cites.
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh · 2015
Earlier work this paper cites.
Multiple object recognition with visual attention
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu · 2015
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Deep convolutional inverse graphics network
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Joshua B Tenenbaum · 2015
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Minh-Thang Luong, Hieu Pham, and Christopher D Manning · 2015
Earlier work this paper cites.
Grammar as a foreign language
Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton · 2015
Earlier work this paper cites.
Weakly-supervised disentangling with recurrent transformations for 3d view synthesis
Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee · 2015
Earlier work this paper cites.
Learning to compose neural networks for question answering
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein · 2016
Earlier work this paper cites.

Similar

Attend, infer, repeat: Fast scene understanding with generative models
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton · 2016
Cited alongside, same era.
Revisiting visual question answering baselines
Allan Jabri, Armand Joulin, and Laurens van der Maaten · 2016
Cited alongside, same era.
The malmo platform for artificial intelligence experimentation
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh · 2016
Cited alongside, same era.
Neural programmer: Inducing latent programs with gradient descent
Arvind Neelakantan, Quoc V Le, and Ilya Sutskever · 2016
Cited alongside, same era.
Stacked attention networks for image question answering
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola · 2016
Cited alongside, same era.
Deepcoder: Learning to write programs
Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow · 2017
Cited alongside, same era.
Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation
Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong · 2017
Cited alongside, same era.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh · 2017
Cited alongside, same era.
From language to programs: Bridging reinforcement learning and maximum marginal likelihood
Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang · 2017
Cited alongside, same era.
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Cited alongside, same era.
Learning to reason: End-to-end module networks for visual question answering
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko · 2017
Cited alongside, same era.

Then

Learning disentangled representations with semi-supervised deep generative models
N Siddharth, T. B. Paige, J.W. Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr · 2017
Later among the works it cites.
Explicit knowledge-based reasoning for visual question answering
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick · 2017
Later among the works it cites.
Neural scene de-rendering
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli · 2017
Later among the works it cites.
Structured attentions for visual question answering
Chen Zhu, Yanpeng Zhao, Shuaiyi Huang, Kewei Tu, and Yi Ma · 2017
Later among the works it cites.
Explicit reasoning over end-to-end neural architectures for visual question answering
Somak Aditya, Yezhou Yang, and Chitta Baral · 2018
Closest in time.
Learning interpretable spatial operations in a rich 3d blocks world
Yonatan Bisk, Kevin J Shih, Yejin Choi, and Daniel Marcu · 2018
Closest in time.
Visual question reasoning on general dependency tree
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin · 2018
Closest in time.
Detectron
Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He · 2018
Closest in time.
Weakly-supervised semantic parsing with abstract examples
Omer Goldman, Veronica Latcinnik, Udi Naveh, Amir Globerson, and Jonathan Berant · 2018
Closest in time.
Scan: learning abstract hierarchical compositional visual concepts
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner · 2018
Closest in time.
Compositional attention networks for machine reasoning
Drew A Hudson and Christopher D Manning · 2018
Closest in time.
Transparency by design: Closing the gap between performance and interpretability in visual reasoning
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar · 2018
Closest in time.
Learning by asking questions
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten · 2018
Closest in time.
Film: Visual reasoning with a general conditioning layer
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville · 2018
Closest in time.
Ddrprog: A clevr differentiable dynamic reasoning programmer
Original
Joseph Suarez, Justin Johnson, and Fei-Fei Li · 2018
Closest in time.
Generative models of visually grounded imagination
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…