Benchmarking Vision Language Models for Cultural Understanding

Extracting cultural commonsense knowledge at scale

Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023 · 1917

Earlier work this paper cites.

Cultures and organizations: software of the mind: intercultural cooperation and its importance for survival , 3rd edition

Geert Hofstede, Gert Jan Hofstede, and Michael Minkov. 2010 · 2010

Earlier work this paper cites.

Microsoft COCO: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014 · 2014

Earlier work this paper cites.

Google lens api

Google. 2017 · 2017

Earlier work this paper cites.

Data statements for natural language processing: Toward mitigating system bias and enabling better science

Emily M. Bender and Batya Friedman. 2018 · 2018

Earlier work this paper cites.

Multimodal C4: An open, billion-scale corpus of images interleaved with text

Original

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. 2023 · 2018

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020

Earlier work this paper cites.

Google landmarks dataset v2 – a large-scale benchmark for instance-level recognition and retrieval

T. Weyand, A. Araujo, B. Cao, and J. Sim. 2020 · 2020

Earlier work this paper cites.

CLIPScore: A reference-free evaluation metric for image captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021 · 2021

Earlier work this paper cites.

Visually grounded reasoning across languages and cultures

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021 · 2021

Earlier work this paper cites.

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 · 2021

Earlier work this paper cites.

Broaden the vision: Geo-diverse visual commonsense reasoning

Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. 2021 · 2021

Earlier work this paper cites.

IGLUE: A benchmark for transfer learning across modalities, tasks, and languages

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. 2022 · 2022

Earlier work this paper cites.

Scaling instruction-finetuned language models

Original

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022

Earlier work this paper cites.

The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world

William Gaviria Rojas, Sudnya Diamos, Keertan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022 · 2022

Cited alongside, same era.

World Values Survey: Round seven - country-pooled datafile version 3.0

Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Marta Lagos, Juan Diez-Medrano, Pippa Norris, Eduard Ponarin, and Bi Puranen. 2022 · 2022

Cited alongside, same era.

Challenges and strategies in cross-cultural NLP

Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022 · 2022

Cited alongside, same era.

Crossmodal-3600: A massively multilingual multimodal evaluation dataset

Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022 · 2022

Cited alongside, same era.

MaXM: Towards multilingual visual question answering

Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023 · 2023

Evaluating visual and cultural interpretation: The k-viscuit benchmark with human-vlm collaboration

Original

Yujin Baek, ChaeHun Park, Jaeseok Kim, Yu-Jung Heo, Du-Seong Chang, and Jaegul Choo. 2024 · 2024

Closest in time.

Paligemma: A versatile 3b vlm for transfer

Original

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. 2024 · 2024

Closest in time.

From local concepts to universals: Evaluating the multicultural understanding of vision-language models

Original

Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, and Vered Shwartz. 2024 · 2024

Closest in time.

How culturally aware are vision-language models?

Original