CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

A Step Toward Quantifying Independently Reproducible Machine Learning Research, September 2019

Original

Edward Raff · 1909

Earlier work this paper cites.

Prediction Interval: What to Expect When You’re Expecting … A Replication

Jeffrey R. Spence and David J. Stanley · 1932

Earlier work this paper cites.

Push button replication: Is impact evaluation evidence for international development verifiable?

Benjamin D. K. Wood, Rui Müller, and Annette N. Brown · 1932

Earlier work this paper cites.

WaveLab and Reproducible Research

Jonathan B. Buckheit and David L. Donoho · 1995

Earlier work this paper cites.

Lessons from the JMCB Archive

B. D. McCullough, Kerry Anne McGeary, and Teresa D. Harrison · 2006

Earlier work this paper cites.

Repeatability of published microarray gene expression analyses

John P. A. Ioannidis, David B. Allison, Catherine A. Ball, Issa Coulibaly, Xiangqin Cui, Aedín C. Culhane, Mario Falchi, Cesare Furlanello, Laurence Game, Giuseppe Jurman, Jon Mangion, Tapan Mehta, Michael Nitzberg, Grier P. Page, Enrico Petretto, and Vera van Noort · 2009

Earlier work this paper cites.

Recommendations for utilizing and reporting population genetic analyses: the reproducibility of genetic clustering using the program structure

Kimberly J. Gilbert, Rose L. Andrew, Dan G. Bock, Michelle T. Franklin, Nolan C. Kane, Jean-Sébastien Moore, Brook T. Moyers, Sébastien Renaut, Diana J. Rennison, Thor Veen, and Timothy H. Vines · 2012

Earlier work this paper cites.

Assessing the reproducibility of discriminant function analyses

Rose L. Andrew, Arianne Y.K. Albert, Sebastien Renaut, Diana J. Rennison, Dan G. Bock, and Tim Vines · 2015

Earlier work this paper cites.

Repeatability in computer systems research

Christian Collberg and Todd A. Proebsting · 2016

Earlier work this paper cites.

Computational Reproducibility via Containers in Psychology

April Clyburne-Sherin, Xu Fei, and Seth Ariel Green · 2018

Earlier work this paper cites.

How to make replication the norm

Paul Gertler, Sebastian Galiani, and Mauricio Romero · 2018

Earlier work this paper cites.

Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition

Tom E. Hardwicke, Maya B. Mathur, Kyle MacDonald, Gustav Nilsonne, George C. Banks, Mallory C. Kidwell, Alicia Hofelich Mohr, Elizabeth Clayton, Erica J. Yoon, Michael Henry Tessler, Richie L. Lenne, Sara Altman, Bria Long, and Michael C. Frank · 2018

Earlier work this paper cites.

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Markus Konkol, Christian Kray, and Max Pfeiffer · 2018

Earlier work this paper cites.

Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine

Florian Naudet, Charlotte Sakarovitch, Perrine Janiaud, Ioana Cristea, Daniele Fanelli, David Moher, and John P. A. Ioannidis · 2018

Earlier work this paper cites.

Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature

Daniel Stockemer, Sebastian Koehler, and Tobias Lentz · 2018

Earlier work this paper cites.

Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge

David M. Liu and Matthew J. Salganik · 2019

Earlier work this paper cites.

Reproducibility and Replicability in Science

National Academies of Sciences Engineering and Medicine · 2019

Earlier work this paper cites.

Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology

Pepijn Obels, Daniël Lakens, Nicholas A. Coles, Jaroslav Gottfried, and Seth A. Green · 2020

Earlier work this paper cites.

A Systematic Review of Reproducibility Research in Natural Language Processing

Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter · 2021

Cited alongside, same era.

Evaluating Large Language Models Trained on Code, July 2021

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021

Cited alongside, same era.

Improving Reproducibility in Machine Learning Research(A Report from the NeurIPS 2019 Reproducibility Program)

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle · 2021

Cited alongside, same era.

AI and the Everything in the Whole Wide World Benchmark, 2021

Security of AI Agents, 2024

Original

Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen · 2024

Closest in time.

AI Agents That Matter, July 2024

Original

Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan · 2024

Closest in time.

Everyone’s talking about Sakana’s AI scientist. But no-one’s answering the big question: is its output good?, August 2024

Jimmy Koppel · 2024

Closest in time.

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, August 2024

Original

Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha · 2024

Closest in time.

OpenEQA: Embodied Question Answering in the Era of Foundation Models

Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran · 2024