Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b · 1905

Earlier work this paper cites.

Deepreid: Deep filter pairing neural network for person re-identification

Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. 2014 · 2014

Earlier work this paper cites.

XNLI: Evaluating cross-lingual sentence representations

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018

Earlier work this paper cites.

The natural language decathlon: Multitask learning as question answering

Original

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018 · 2018

Earlier work this paper cites.

CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

Rohit Girdhar and Deva Ramanan. 2020 · 2020

Earlier work this paper cites.

Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2020

Earlier work this paper cites.

Measuring compositional generalization: A comprehensive method on realistic data

Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. 2020 · 2020

Earlier work this paper cites.

COGS: A compositional generalization challenge based on semantic interpretation

Najoung Kim and Tal Linzen. 2020 · 2020

Earlier work this paper cites.

Guided generation of cause and effect

Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2021 · 2021

Earlier work this paper cites.

Dall·e: Creating images from text

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021 · 2021

Earlier work this paper cites.

COM2SENSE: A commonsense reasoning benchmark with complementary sentences

Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-Lin Wu, Xuezhe Ma, and Nanyun Peng. 2021 · 2021

Earlier work this paper cites.

Adversarial glue: A multi-task benchmark for robustness evaluation of language models

Original

Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021 · 2021

Earlier work this paper cites.

Acre: Abstract causal reasoning beyond covariation

Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. 2021 · 2021

Earlier work this paper cites.

e-care: a new dataset for exploring explainable causal reasoning

Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022 · 2022

Earlier work this paper cites.

CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models

Jörg Frohberg and Frank Binder. 2022 · 2022

Earlier work this paper cites.

The abduction of sherlock holmes: A dataset for visual abductive reasoning

Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022 · 2022

Earlier work this paper cites.

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Original

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022 · 2022

Earlier work this paper cites.

Openflamingo: An open-source framework for training large autoregressive vision-language models

Original

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023 · 2023

Cited alongside, same era.

Instructblip: Towards general-purpose vision-language models with instruction tuning

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023 · 2023

Cited alongside, same era.

Cladder: Assessing causal reasoning in language models

Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, LYU Zhiheng, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, et al. 2023 · 2023

Cited alongside, same era.

Causal reasoning and large language models: Opening a new frontier for causality

Original

Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. 2023 · 2023

Cited alongside, same era.

Gemini 1.5 models

DeepMind. 2024 · 2024

Closest in time.

Introducing flux.1 tools

FLUXAI. 2024 · 2024

Closest in time.

Blink: Multimodal large language models can see but not perceive

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024 · 2024

Closest in time.

Lcm-lookahead for encoder-based text-to-image personalization

Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2024 · 2024

Closest in time.

Visual riddles: a commonsense and world knowledge challenge for large vision and language models

Nitzan Bitton Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. 2024 · 2024

Closest in time.

Can large language models infer causation from correlation?

Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. 2024 · 2024