Benchmarking Distributional Alignment of Large Language Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Markov chains and mixing times

David A. Levin, Yuval Peres, and Elizabeth L. Wilmer. 2006 · 2006

Earlier work this paper cites.

The neglected 95%: why american psychology needs to become less american

Jeffrey Jensen Arnett. 2008 · 2008

Earlier work this paper cites.

(Mis)perceptions of Partisan Polarization in the American Public

Matthew S. Levendusky and Neil Malhotra. 2015 · 2015

Earlier work this paper cites.

Measuring public opinion with surveys

Adam Berinsky. 2017 · 2017

Earlier work this paper cites.

Deep reinforcement learning from human preferences

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017

Earlier work this paper cites.

On calibration of modern neural networks

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017 · 2017

Earlier work this paper cites.

The perception gap: How false impressions are pulling americans apart

Daniel A Yudkin, Stephen Hawkins, and Tim Dixon. 2019 · 2019

Earlier work this paper cites.

Beyond weird: A review of the last decade and a look ahead to the global laboratory of the future

Coren Apicella, Ara Norenzayan, and Joseph Henrich. 2020 · 2020

Earlier work this paper cites.

Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs

Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W. Duncan Wadsworth, and Hanna Wallach. 2021 · 2021

Earlier work this paper cites.

Simcse: Simple contrastive learning of sentence embeddings

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021 · 2021

Earlier work this paper cites.

Consequences of asking sensitive questions in surveys

Ting Yan. 2021 · 2021

Earlier work this paper cites.

CommunityLM: Probing partisan worldviews from language models

Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022 · 2022

Earlier work this paper cites.

Language models (mostly) know what they know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022 · 2022

Earlier work this paper cites.

Surfacing racial stereotypes through identity portrayal

Gauri Kambhatla, Ian Stewart, and Rada Mihalcea. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

Using large language models to simulate multiple humans and replicate human subject studies

Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023 · 2023

Earlier work this paper cites.

Out of one, many: Using language models to simulate human samples

Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023 · 2023

Earlier work this paper cites.

CoMPosT: Characterizing and evaluating caricature in LLM simulations

Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023b · 2023

Earlier work this paper cites.

Can ai language models replace human participants?

Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023 · 2023

Earlier work this paper cites.

Alpacafarm: A simulation framework for methods that learn from human feedback

Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023 · 2023

Earlier work this paper cites.

Ai and the transformation of social science research

Igor Grossmann, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham. 2023 · 2023

Earlier work this paper cites.

Large language models as simulated economic agents: What can we learn from homo silicus?

John J Horton. 2023 · 2023

Earlier work this paper cites.

Prompting is not a substitute for probability measurements in large language models

Jennifer Hu and Roger P. Levy. 2023 · 2023

Earlier work this paper cites.

Aligning language models to user opinions

EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023 · 2023

Earlier work this paper cites.

Large language models as superpositions of cultural perspectives

Original

Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023 · 2023

Cited alongside, same era.

Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness?

Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. 2023 · 2023

Cited alongside, same era.

Discovering language model behaviors with model-written evaluations

Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. 2023 · 2023

Cited alongside, same era.

Whose opinions do language models reflect?

How random is random? evaluating the randomness and humaness of llms’ coin flips

Original

Katherine Van Koevering and Jon Kleinberg. 2024 · 2024

Closest in time.

Benchmarking cognitive biases in large language models as evaluators

Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024 · 2024

Closest in time.

The steerability of large language models toward data-driven personas

Junyi Li, Charith Peris, Ninareh Mehrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. 2024a · 2024

Closest in time.

Evaluating large language model biases in persona-steered generation

Andy Liu, Mona Diab, and Daniel Fried. 2024 · 2024

Closest in time.

Beyond probabilities: Unveiling the misalignment in evaluating large language models

Chenyang Lyu, Minghao Wu, and Alham Aji. 2024 · 2024

Closest in time.