2017

Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

Li, Guohao, Su, Hang, Zhu, Wenwu

Understand

Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language.

Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents.
To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks.
Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures.

Built on

Dbpedia: A nucleus for a web of open data
S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives · 2007
Earlier work this paper cites.
Freebase: a collaboratively created graph database for structuring human knowledge
K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor · 2008
Earlier work this paper cites.
Natural language processing with Python: analyzing text with the natural language toolkit
S. Bird, E. Klein, and E. Loper · 2009
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Representing general relational knowledge in ConceptNet 5
R. Speer and C. Havasi · 2012
Earlier work this paper cites.
Translating embeddings for modeling multi-relational data
A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko · 2013
Earlier work this paper cites.
Glove: Global vectors for word representation
J. Pennington, R. Socher, and C. Manning · 2014
Earlier work this paper cites.
Memory networks
Original
J. Weston, S. Chopra, and A. Bordes · 2014
Earlier work this paper cites.
Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Earlier work this paper cites.
Large-scale simple question answering with memory networks
Original
A. Bordes, N. Usunier, S. Chopra, and J. Weston · 2015
Earlier work this paper cites.
Are you talking to a machine? dataset and methods for multilingual image question
H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu · 2015
Earlier work this paper cites.
Visual turing test for computer vision systems
D. Geman, S. Geman, N. Hallonquist, and L. Younes · 2015
Earlier work this paper cites.

Similar

Fast r-cnn
R. Girshick · 2015
Cited alongside, same era.
Ask your neurons: A neural-based approach to answering questions about images
M. Malinowski, M. Rohrbach, and M. Fritz · 2015
Cited alongside, same era.
Image question answering: A visual semantic embedding model and a new dataset
M. Ren, R. Kiros, and R. Zemel · 2015
Cited alongside, same era.
End-to-end memory networks
S. Sukhbaatar, J. Weston, R. Fergus, et al · 2015
Cited alongside, same era.
Explicit knowledge-based reasoning for visual question answering
Original
P. Wang, Q. Wu, C. Shen, A. v. d. Hengel, and A. Dick · 2015
Cited alongside, same era.
Multimodal compact bilinear pooling for visual question answering and visual grounding
Original
A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Ask me anything: Dynamic memory networks for natural language processing
A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, V. Zhong, R. Paulus, and R. Socher · 2016
Cited alongside, same era.
Hierarchical question-image co-attention for visual question answering
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Cited alongside, same era.
Ask me anything: Free-form visual question answering based on knowledge from external sources
Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2016
Cited alongside, same era.
Dynamic memory networks for visual and textual question answering
C. Xiong, S. Merity, and R. Socher · 2016
Cited alongside, same era.

Then

Ask, attend and answer: Exploring question-guided spatial attention for visual question answering
H. Xu and K. Saenko · 2016
Later among the works it cites.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola · 2016
Later among the works it cites.
Visual7W: Grounded Question Answering in Images
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Bottom-up and top-down attention for image captioning and vqa
Original
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2017
Closest in time.
Visual question answering: Datasets, algorithms, and future challenges
K. Kafle and C. Kanan · 2017
Closest in time.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al · 2017
Closest in time.
Visual question answering with memory-augmented networks
Original
C. Ma, C. Shen, A. Dick, and A. v. d. Hengel · 2017
Closest in time.
Memen: Multi-layer embedding with memory networks for machine comprehension
Original
B. Pan, H. Li, Z. Zhao, B. Cao, D. Cai, and X. He · 2017
Closest in time.
Fvqa: fact-based visual question answering
P. Wang, Q. Wu, C. Shen, A. Dick, and A. van den Hengel · 2017
Closest in time.
Visual question answering: A survey of methods and datasets
Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2017
Closest in time.
Multi-level attention networks for visual question answering
D. Yu, J. Fu, T. Mei, and Y. Rui · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…