CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005

Earlier work this paper cites.

Language models as knowledge bases?

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019 · 2019

Earlier work this paper cites.

ProtoQA: A question answering dataset for prototypical common-sense reasoning

Michael Boratko, Xiang Li, Tim O’Gorman, Rajarshi Das, Dan Le, and Andrew McCallum. 2020 · 2020

Earlier work this paper cites.

Dynabench: Rethinking benchmarking in nlp

Original

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021 · 2021

Earlier work this paper cites.

Why exposure bias matters: An imitation learning perspective of error accumulation in language generation

Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. 2022 · 2022

Earlier work this paper cites.

Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned

Original

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022 · 2022

Earlier work this paper cites.

Challenges and strategies in cross-cultural nlp

Original

Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, et al. 2022 · 2022

Earlier work this paper cites.

Language models (mostly) know what they know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022 · 2022

Earlier work this paper cites.

Probing pre-trained language models for cross-cultural differences in values

Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023 · 2023

Earlier work this paper cites.

Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study

Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023 · 2023

Earlier work this paper cites.

Eticor: Corpus for analyzing llms for etiquettes

Original

Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023 · 2023

Earlier work this paper cites.

Multilingual language models are not multicultural: A case study in emotion

Original

Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023 · 2023

Earlier work this paper cites.

Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models

Original

Amr Keleg and Walid Magdy. 2023 · 2023

Earlier work this paper cites.

Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings

Original

Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023 · 2023

Cited alongside, same era.

A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

Original

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2023 · 2023

Cited alongside, same era.

Having beer after prayer? measuring cultural bias in large language models

Original

Tarek Naous, Michael Joseph Ryan, and Wei Xu. 2023 · 2023

Cited alongside, same era.

Extracting cultural commonsense knowledge at scale

Tuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, and Gerhard Weikum. 2022 · 2023

Cited alongside, same era.

FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models

Chatbot arena: An open platform for evaluating llms by human preference

Original

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024 · 2024

Closest in time.

Bertaqa: How much do language models know about local culture?

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, and Mikel Artetxe. 2024 · 2024

Closest in time.

Massively multi-cultural knowledge acquisition & lm benchmarking

Original

Yi Ren Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024 · 2024

Closest in time.

Culturellm: Incorporating cultural differences into large language models

Original

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024 · 2024

Closest in time.

Best practices and lessons learned on synthetic data for language models