Fetching the paper…

Reading the bibliography…

2024

A Surprising Failure? Multimodal LLMs and the NLVR Challenge

Wu, Anne, Brantley, Kianté, Artzi, Yoav

Original
PDF
Post on X

Understand

This study evaluates three state-of-the-art MLLMs -- GPT-4V, Gemini Pro, and the open-source model IDEFICS -- on the compositional natural language vision reasoning task NLVR.

Given a human-written sentence paired with a synthetic image, this task requires the model to determine the truth value of the sentence with respect to the image.
Despite the strong performance demonstrated by these models, we observe they perform poorly on NLVR, which was constructed to require compositional and spatial reasoning, and to be robust for semantic and systematic biases.

A Surprising Failure? Multimodal LLMs and the NLVR Challenge · Around

Built on

C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset
Original
Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh · 2017
Earlier work this paper cites.
A corpus of natural language for visual reasoning
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi · 2017
Earlier work this paper cites.
Visual referring expression recognition: What do systems actually learn?
Original
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirkpatrick · 2018
Earlier work this paper cites.
What is learned in visually grounded neural syntax acquisition
Original
Noriyuki Kojima, Hadar Averbuch-Elor, Alexander M Rush, and Yoav Artzi · 2020
Earlier work this paper cites.

Similar

Km4: Visual reasoning via knowledge embedding memory model with mutual modulation
Original
Wenbo Zheng, Lan Yan, Chao Gou, and Fei-Yue Wang · 2021
Cited alongside, same era.
Flamingo: a visual language model for few-shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al · 2022
Cited alongside, same era.
Qlora: Efficient finetuning of quantized llms
Original
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer · 2023
Cited alongside, same era.
Gemini: a family of highly capable multimodal models
Original
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al · 2023
Cited alongside, same era.

Then

Obelisc: An open web-scale filtered dataset of interleaved image-text documents
Original
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al · 2023
Later among the works it cites.
Gpt-4v(ision) system card
OpenAI · 2023
Later among the works it cites.
The dawn of lmms: Preliminary explorations with gpt-4v (ision)
Original
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…