Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Visual entailment: A novel task for fine-grained image understanding

Original

Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019 · 1901

Earlier work this paper cites.

Albert: A lite bert for self-supervised learning of language representations

Original

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019 · 1909

Earlier work this paper cites.

All you may need for VQA are image captions

Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022 · 1963

Earlier work this paper cites.

Multimodal dialogue state tracking by qa approach with data augmentation

Original

Xiangyang Mou, Brandyn Sigouin, Ian Steenstra, and Hui Su. 2020 · 2007

Earlier work this paper cites.

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014 · 2014

Earlier work this paper cites.

Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015 · 2015

Earlier work this paper cites.

Foil it! find one mismatch between image and language caption

Original

Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017 · 2017

Earlier work this paper cites.

Multimodal continuous emotion recognition with data augmentation using recurrent neural networks

Jian Huang, Ya Li, Jianhua Tao, Zheng Lian, Mingyue Niu, and Minghao Yang. 2018 · 2018

Earlier work this paper cites.

Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018 · 2018

Earlier work this paper cites.

Data augmentation techniques for the video question answering task

Alex Falcon, Oswald Lanz, and Giuseppe Serra. 2020 · 2020

Earlier work this paper cites.

Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021 · 2021

Earlier work this paper cites.

Kat: A knowledge augmented transformer for vision-and-language

Original

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2021 · 2021

Earlier work this paper cites.

What makes good in-context examples for gpt- 3 3 ?

Original

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Learning to retrieve prompts for in-context learning

Original

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021 · 2021

Earlier work this paper cites.

Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021 · 2021

Earlier work this paper cites.

Multimodal few-shot learning with frozen language models

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021 · 2021

Earlier work this paper cites.

Cm3: A causal masked multimodal model of the internet

Original

Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022 · 2022

Earlier work this paper cites.

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

Revive: Regional visual representation matters in knowledge-based visual question answering

Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022 · 2022

Earlier work this paper cites.

What makes good in-context examples for GPT-3?

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022 · 2022

Earlier work this paper cites.

Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022 · 2022

Earlier work this paper cites.

Rethinking the role of demonstrations: What makes in-context learning work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Cited alongside, same era.

VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena

Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2022 · 2022

Cited alongside, same era.

Pre-training image-language transformers for open-vocabulary tasks

Original

AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. 2022 · 2022

Cited alongside, same era.

Laion-5b: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022 · 2022

Cited alongside, same era.

Self-consistency improves chain of thought reasoning in language models

Original

Llama: Open and efficient foundation language models (2023)

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Later among the works it cites.

mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

Original

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023 · 2023

Later among the works it cites.

Sigmoid loss for language image pre-training

Original

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023 · 2023

Later among the works it cites.

Multimodal chain-of-thought reasoning in language models

Original

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023 · 2023

Later among the works it cites.