Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation

Domenic V Cicchetti · 1991

Earlier work this paper cites.

Examining the value added by committee discussion in the review of applications for research awards

Michael Obrecht, Karl Tibelius, and Guy D’Aloisio · 2007

Earlier work this paper cites.

A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants

Lutz Bornmann, Rüdiger Mutz, and Hans-Dieter Daniel · 2010

Earlier work this paper cites.

Panel discussion does not improve reliability of peer review for medical research grant proposals

Mikael Fogelholm, Saara Leppinen, Anssi Auvinen, Jani Raitanen, Anu Nuutinen, and Kalervo Väänänen · 2012

Earlier work this paper cites.

The NIPS Experiment

Neil D. Lawrence · 2014

Earlier work this paper cites.

The NIPS experiment

Eric Price · 2014

Cited alongside, same era.

The NIPS experiment

John Langford · 2015

Cited alongside, same era.

Peer review and the NIPS experiment

Neil D. Lawrence · 2015

Cited alongside, same era.

Your comments are meaner than your score: score calibration talk influences intra-and inter-panel variability during scientific grant peer review

Elizabeth Pier, Joshua Raclaw, Anna Kaatz, Markus Brauer, Molly Carnes, Mitchell Nathan, and Cecilia Ford · 2017

Cited alongside, same era.

A retrospective on the NeurIPS 2021 ethics review process

Samy Bengio, Inioluwa Deborah Raji, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan · 2021

Cited alongside, same era.

Introducing the NeurIPS 2021 paper checklist

Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan · 2021

Later among the works it cites.

The NeurIPS 2021 consistency experiment

Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan · 2021

Later among the works it cites.

Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment

Original

Corinna Cortes and Neil D. Lawrence · 2021

Later among the works it cites.

Noise: A Flaw in Human Judgment

Daniel Kahneman, Olivier Sibony, and Cass R Sunstein · 2021

Later among the works it cites.

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

Original

Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, Jennifer Wortman Vaughan, Zhenyu Xue, Hal Daumé III, Emma Pierson, and Nihar B. Shah · 2022

Later among the works it cites.