2020

Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

Pineau, Joelle, Vincent-Lamarre, Philippe, Sinha, Koustuv et al.

Understand

One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable.

Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings.
Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice.
Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors.

Built on

The Scientific Method in the Science of Machine Learning
Original
Jessica Zosa Forde and Michela Paganini · 1904
Earlier work this paper cites.
Fermat’s Last Theorem
Wikipedia · 2003
Earlier work this paper cites.
The need for open source software in machine learning
SÃķren Sonnenburg, Mikio L Braun, Cheng Soon Ong, Samy Bengio, Leon Bottou, Geoffrey Holmes, Yann LeCun, Klaus-Robert MÃžller, Fernando Pereira, Carl Edward Rasmussen, et al · 2007
Earlier work this paper cites.
The repeatability experiment of sigmod 2008
Ioana Manolescu, Loredana Afanasiev, Andrei Arion, Jens Dittrich, Stefan Manegold, Neoklis Polyzotis, Karl Schnaitter, Pierre Senellart, Spyros Zoupanos, and Dennis Shasha · 2008
Earlier work this paper cites.
Checklist manifesto, the (HB)
Atul Gawande · 2010
Earlier work this paper cites.
Repeatability and workability evaluation of sigmod 2011
Philippe Bonnet, Stefan Manegold, Matias Bjørling, Wei Cao, Javier Gonzalez, Joel Granados, Nancy Hall, Stratos Idreos, Milena Ivanova, Ryan Johnson, David Koop, Tim Kraska, René Müller, Dan Olteanu, Paolo Papotti, Christine Reilly, Dimitris Tsirogiannis, Cong Yu, Juliana Freire, and Dennis Shasha · 2011
Earlier work this paper cites.
Consolidated standards of reporting trials (consort) and the completeness of reporting of randomised controlled trials (rcts) published in medical journals
Lucy Turner, Larissa Shamseer, Douglas G Altman, Laura Weeks, Jodi Peters, Thilo Kober, Sofia Dias, Kenneth F Schulz, Amy C Plint, and David Moher · 2012
Earlier work this paper cites.
Journals unite for reproducibility
Marcia McNutt · 2014
Earlier work this paper cites.
Back to basics: Checklists in aviation and healthcare
Robyn Clay-Williams and Lacey Colligan · 2015
Earlier work this paper cites.
Promoting an open research culture
Brian A Nosek, George Alter, George C Banks, Denny Borsboom, Sara D Bowman, Steven J Breckler, Stuart Buck, Christopher D Chambers, Gilbert Chin, Garret Christensen, et al · 2015
Earlier work this paper cites.
1,500 scientists lift the lid on reproducibility
Monya Baker · 2016
Earlier work this paper cites.
Toward the geoscience paper of the future: Best practices for documenting and sharing research from data to software to provenance
Yolanda Gil, Cédric H. David, Ibrahim Demir, Bakinam T. Essawy, Robinson W. Fulweiler, Jonathan L. Goodall, Leif Karlstrom, Huikyo Lee, Heath J. Mills, Ji-Hyun Oh, Suzanne A. Pierce, Allen Pope, Mimi W. Tzeng, Sandra R. Villamizar, and Xuan Yu · 2016
Earlier work this paper cites.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Earlier work this paper cites.
Reproducibility in Machine Learning Workshop
ICML · 2017
Earlier work this paper cites.
On the state of the art of evaluation in neural language models
Original
Gábor Melis, Chris Dyer, and Phil Blunsom · 2017
Earlier work this paper cites.

Similar

A manifesto for reproducible science
Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie Du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis · 2017
Cited alongside, same era.
Reproducibility ’17: Proceedings of the reproducibility workshop
SIGCOMM · 2017
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
State of the art: Reproducibility in artificial intelligence
Odd Erik Gundersen and Sigbjørn Kjensmo · 2018
Cited alongside, same era.
On reproducible ai: Towards reproducible research, open science and digital scholarship in ai publications
Odd Erik Gundersen, Yolanda Gil, and David W Aha · 2018
Cited alongside, same era.
Deep reinforcement learning that matters
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger · 2018
Cited alongside, same era.
ICLR 2018 Reproducibility Challenge
ICLR · 2018
Cited alongside, same era.
Reproducibility in Machine Learning Workshop
ICML · 2018
Cited alongside, same era.
Are gans created equal? a large-scale study
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet · 2018
Cited alongside, same era.
Science journals: Editorial policies., 2018
Science - AAAS · 2018
Cited alongside, same era.
Open science now: A systematic literature review for an integrated definition
Rubén Vicente-Sáez and Clara Martínez-Fuentes · 2018
Cited alongside, same era.
In Proceedings of the 36th International Conference on Machine Learning , 2019
Elf opengo: an analysis and open reimplementation of alphazero · 2019
Cited alongside, same era.

Then

Reproducibility in Machine Learning Workshop
ICLR · 2019
Later among the works it cites.
Call for papers
ICML · 2019
Later among the works it cites.
Out-of-the-box reproducibility: A survey of machine learning platforms
Ricahrd Isdahl and Odd Erik Gundersen · 2019
Later among the works it cites.
Many labs 4: Failure to replicate mortality salience effect with and without original author involvement
Richard A Klein, Corey L Cook, Charles R Ebersole, Christine Vitiello, Brian A Nosek, Christopher R Chartier, Cody D Christopherson, Samuel Clay, Brian Collisson, Jarret Crawford, et al · 2019
Later among the works it cites.
Extension of the consort and spirit statements
Xiaoxuan Liu, Livia Faes, Melanie J Calvert, and Alastair K Denniston · 2019
Later among the works it cites.
Reproducibility and replicability in science
National Academies of Sciences, Engineering, and Medicine · 2019
Later among the works it cites.
Neurips 2019 reproducibility challenge, 2021
OpenReview.net · 2019
Later among the works it cites.
ICLR Reproducibility Challenge 2019
Joelle Pineau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle · 2019
Later among the works it cites.
A step toward quantifying independently reproducible machine learning research
Edward Raff · 2019
Later among the works it cites.
Call for Reproducibility papers
ECIR · 2020
Closest in time.
The rescience journal. reproducible science is good. replicated science is better., 2020
ReScience C · 2020
Closest in time.
NeurIPS 2019 Reproducibility Challenge
Koustuv Sinha, Joelle Pineau, Jessica Forde, Rosemary Nan Ke, and Hugo Larochelle · 2020
Closest in time.
Nature checklist
Nature Checklist · 2021
Closest in time.
Reporting standards and availability of data, materials, code and protocols, 2021
Nature Research · 2021
Closest in time.
Enhancing the quality and transparency of health research., 2021
The EQUATOR Network · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…