A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts

Measurement validity: A shared standard for qualitative and quantitative research

Robert Adcock and David Collier · 2001

Earlier work this paper cites.

Show Your Work: Improved Reporting of Experimental Results

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith · 2019

Earlier work this paper cites.

Dissecting racial bias in an algorithm used to manage the health of populations

Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan · 2019

Earlier work this paper cites.

Problem formulation and fairness

Samir Passi and Solon Barocas · 2019

Earlier work this paper cites.

Counterfactual risk assessments, evaluation, and fairness

Amanda Coston, Alan Mishler, Edward H Kennedy, and Alexandra Chouldechova · 2020

Earlier work this paper cites.

Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach · 2021

Earlier work this paper cites.

Hyperparameter Optimization Is Deceiving Us, and How to Stop It

A. Feder Cooper, Yucheng Lu, Jessica Forde, and Christopher M De Sa · 2021

Earlier work this paper cites.

Proxies: The cultural work of standing in

Dylan Mulvin · 2021

Cited alongside, same era.

Are my deep learning systems fair? an empirical study of fixed-seed training

Shangshu Qian, Hung Viet Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah · 2021

Cited alongside, same era.

Quantifying memorization across neural language models

Original

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang · 2022

Cited alongside, same era.

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy, 2023

Original

Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, and Nicholas Carlini · 2023

Cited alongside, same era.

Optimization’s neglected normative commitments

Benjamin Laufer, Thomas Gilbert, and Helen Nissenbaum · 2023

Cited alongside, same era.

The Files are in the Computer: Copyright, Memorization, and Generative AI

Original

A Feder Cooper and James Grimmelmann · 2024

Closest in time.

Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification

A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grimmelmann, Jon Kleinberg, Siddhartha Sen, and Baobao Zhang · 2024

Closest in time.

ECBD: Evidence-centered benchmark design for NLP

Yu Lu Liu, Su Lin Blodgett, Jackie Cheung, Q. Vera Liao, Alexandra Olteanu, and Ziang Xiao · 2024

Closest in time.

The AI index 2024 annual report

Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark · 2024

Closest in time.

On the consistency of hyper-parameter selection in value-based deep reinforcement learning, 2024

Original

Johan Obando-Ceron, João G. M. Araújo, Aaron Courville, and Pablo Samuel Castro · 2024