2021

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

Luo, Man, Zeng, Yankai, Banerjee, Pratyay et al.

Understand

Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images.

One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval.
Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge.
Because of varying knowledge bases, it is hard to fairly compare models' performance.

Built on

Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Visualbert: A simple and performant baseline for vision and language
Original
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019 · 1908
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
WordNet: A lexical database for English
George A. Miller. 1992 · 1992
Earlier work this paper cites.
A framework for learning predictive structures from multiple tasks and unlabeled data
Rie Kubota Ando and Tong Zhang. 2005 · 2005
Earlier work this paper cites.
Scalable training of L1-regularized log-linear models
Galen Andrew and Jianfeng Gao. 2007 · 2007
Earlier work this paper cites.
The probabilistic relevance framework: Bm25 and beyond
S. Robertson and H. Zaragoza. 2009 · 2009
Earlier work this paper cites.
A particle filter algorithm for Bayesian wordsegmentation
Benjamin Börschinger and Mark Johnson. 2011 · 2011
Earlier work this paper cites.
Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa
Original
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2020 · 2012
Earlier work this paper cites.
Learning from 26 languages: Program management and science in the babel program
Mary Harper. 2014 · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014 · 2014
Earlier work this paper cites.
Wikidata: a free collaborative knowledgebase
Denny Vrandečić and Markus Krötzsch. 2014 · 2014
Earlier work this paper cites.
VQA: visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015a · 2015
Earlier work this paper cites.
VQA: visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015b · 2015
Earlier work this paper cites.
Yara parser: A fast and accurate dependency parser
Original
Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015 · 2015
Earlier work this paper cites.
Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing
James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016 · 2016
Earlier work this paper cites.

Similar

Visual7w: Grounded question answering in images
Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. 2016 · 2016
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
Thomas N. Kipf and Max Welling. 2017 · 2017
Cited alongside, same era.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017 · 2017
Cited alongside, same era.
Conceptnet 5.5: An open multilingual graph of general knowledge
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017 · 2017
Cited alongside, same era.
Explicit knowledge-based reasoning for visual question answering
Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel. 2017b · 2017
Cited alongside, same era.
Adaptive document retrieval for deep question answering
Bernhard Kratzwald and Stefan Feuerriegel. 2018 · 2018
Cited alongside, same era.
Ranking paragraphs for improving answer recall in open-domain question answering
Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018 · 2018
Cited alongside, same era.
Out of the box: Reasoning with graph convolution nets for factual visual question answering
Medhini Narasimhan, Svetlana Lazebnik, and Alexander G. Schwing. 2018 · 2018
Cited alongside, same era.
R 3: Reinforced ranker-reader for open-domain question answering
Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018 · 2018
Cited alongside, same era.
Careful selection of knowledge to solve open book question answering
Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. 2019 · 2019
Cited alongside, same era.
Multi-step retriever-reader interaction for scalable open-domain question answering
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.

Then

Towards causal VQA: revealing and reducing spurious correlations by invariant and covariant semantic editing
Vedika Agarwal, Rakshith Shetty, and Mario Fritz. 2020 · 2020
Later among the works it cites.
ConceptBert: Concept-aware representation for visual question answering
François Gardères, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lecue. 2020 · 2020
Later among the works it cites.
MUTANT: A training paradigm for out-of-distribution generalization in visual question answering
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020a · 2020
Later among the works it cites.
Dense passage retrieval for open-domain question answering
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020
Later among the works it cites.
Retrieval-augmented generation for knowledge-intensive NLP tasks
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020 · 2020
Later among the works it cites.
Oscar: Object-semantics aligned pre-training for vision-language tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020 · 2020
Later among the works it cites.
Visuo-lingustic question answering (vlqa) challenge
Shailaja Keyur Sampat, Yezhou Yang, and Chitta Baral. 2020 · 2020
Later among the works it cites.
Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering
Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. 2020 · 2020
Later among the works it cites.
WeaQA: Weak supervision via captions for visual question answering
Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021 · 2021
Closest in time.
Select, substitute, search: A new benchmark for knowledge-augmented visual question answering
Original
Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2021 · 2021
Closest in time.
‘just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of open-ended VQA tasks
Man Luo, Shailaja Keyur Sampat, Riley Tallman, Yankai Zeng, Manuha Vancha, Akarshan Sajja, and Chitta Baral. 2021 · 2021
Closest in time.
Zero-shot neural passage retrieval via domain-targeted synthetic question generation
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021 · 2021
Closest in time.
Natural instructions: Benchmarking generalization to new tasks from natural language instructions
Original
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021 · 2021
Closest in time.
Reasoning over vision and language: Exploring the benefits of supplemental knowledge
Violetta Shevchenko, Damien Teney, Anthony Dick, and Anton van den Hengel. 2021 · 2021
Closest in time.
Multi-modal answer validation for knowledge-based vqa
Original
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…