2021

Cross-Modal Retrieval Augmentation for Multi-Modal Classification

Gur, Shir, Neverova, Natalia, Stauffer, Chris et al.

Understand

Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.

Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA).
First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t.
similar methods.

Built on

Visualbert: A simple and performant baseline for vision and language
Original
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b · 1908
Earlier work this paper cites.
Vl-bert: Pre-training of generic visual-linguistic representations
Original
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019 · 1908
Earlier work this paper cites.
Supervised multimodal bitransformers for classifying images and text
Original
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. 2019 · 1909
Earlier work this paper cites.
Generalization through memorization: Nearest neighbor language models
Original
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019 · 1911
Earlier work this paper cites.
In defense of grid features for visual question answering
Original
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. 2020 · 2001
Earlier work this paper cites.
Realm: Retrieval-augmented language model pre-training
Original
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020 · 2002
Earlier work this paper cites.
Augmenting transformers with knn-based composite memory for dialogue
Original
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2020 · 2004
Earlier work this paper cites.
Dense passage retrieval for open-domain question answering
Original
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2004
Earlier work this paper cites.
Transformer reasoning network for image-text matching and retrieval
Original
Nicola Messina, Fabrizio Falchi, Andrea Esuli, and Giuseppe Amato. 2020b · 2004
Earlier work this paper cites.
Revisiting modulated convolutions for visual counting and beyond
Original
Duy-Kien Nguyen, Vedanuj Goswami, and Xinlei Chen. 2020 · 2004
Earlier work this paper cites.
Are we pretraining it right? digging deeper into visio-linguistic pretraining
Original
Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. 2020b · 2004
Earlier work this paper cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Original
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020 · 2005
Earlier work this paper cites.
Leveraging passage retrieval with generative models for open domain question answering
Original
Gautier Izacard and Edouard Grave. 2020 · 2007
Earlier work this paper cites.
Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge
Original
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. 2020 · 2007
Earlier work this paper cites.
Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders
Original
Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maillet. 2020a · 2008
Earlier work this paper cites.
Neil: Extracting visual knowledge from web data
Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. 2013 · 2013
Earlier work this paper cites.
Learning everything about anything: Webly-supervised visual concept learning
Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. 2014 · 2014
Earlier work this paper cites.

Similar

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014 · 2014
Cited alongside, same era.
Reasoning about object affordances in a knowledge base representation
Yuke Zhu, Alireza Fathi, and Li Fei-Fei. 2014 · 2014
Cited alongside, same era.
Microsoft coco captions: Data collection and evaluation server
Original
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015 · 2015
Cited alongside, same era.
Exploring nearest neighbor approaches for image captioning
Original
Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick. 2015 · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
Andrej Karpathy and Li Fei-Fei. 2015 · 2015
Cited alongside, same era.
Viske: Visual knowledge extraction and question answering by visual verification of relation phrases
Fereshteh Sadeghi, Santosh K Kumar Divvala, and Ali Farhadi. 2015 · 2015
Cited alongside, same era.
Explicit knowledge-based reasoning for visual question answering
Original
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. 2015 · 2015
Cited alongside, same era.
Building a large-scale multimodal knowledge base system for answering visual queries
Original
Yuke Zhu, Ce Zhang, Christopher Ré, and Li Fei-Fei. 2015 · 2015
Cited alongside, same era.
Vse++: Improving visual-semantic embeddings with hard negatives
Original
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017 · 2017
Cited alongside, same era.
Billion-scale similarity search with gpus
Original
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017 · 2017
Cited alongside, same era.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017 · 2017
Cited alongside, same era.
Learning a recurrent residual fusion network for multimodal matching
Yu Liu, Yanming Guo, Erwin M Bakker, and Michael S Lew. 2017 · 2017
Cited alongside, same era.

Then

Pythia v0. 1: the winning entry to the vqa challenge 2018
Original
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018 · 2018
Later among the works it cites.
Stacked cross attention for image-text matching
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018 · 2018
Later among the works it cites.
Out of the box: Reasoning with graph convolution nets for factual visual question answering
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. 2018 · 2018
Later among the works it cites.
Straight to the facts: Learning knowledge base retrieval for factual visual question answering
Medhini Narasimhan and Alexander G Schwing. 2018 · 2018
Later among the works it cites.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018 · 2018
Later among the works it cites.
Fvqa: Fact-based visual question answering
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2018 · 2018
Later among the works it cites.
Deep cross-modal projection learning for image-text matching
Ying Zhang and Huchuan Lu. 2018 · 2018
Later among the works it cites.
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019 · 2019
Later among the works it cites.
Ok-vqa: A visual question answering benchmark requiring external knowledge
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019 · 2019
Later among the works it cites.
Camp: Cross-modal adaptive message passing for text-image retrieval
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. 2019 · 2019
Later among the works it cites.
Uniter: Universal image-text representation learning
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020 · 2020
Later among the works it cites.
Multi-modality cross attention network for image and sentence matching
Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. 2020 · 2020
Later among the works it cites.
Scaling up visual and vision-language representation learning with noisy text supervision
Original
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021 · 2021
Closest in time.
Visualsparta: Sparse transformer fragment-level matching for large-scale text-to-image search
Original
Xiaopeng Lu, Tiancheng Zhao, and Kyusong Lee. 2021 · 2021
Closest in time.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021 · 2021
Closest in time.
Learning fragment self-attention embeddings for image-text matching
Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang. 2019 · 2096
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…