Fetching the paper…
Reading the bibliography…
Background: Many published machine learning studies are irreproducible.
Report of the ACM task force on the core of Computer Science
Peter J Denning, Douglas E Comer, David Gries, Michael C Mulder, Allen Tucker, A Joe Turner, and Paul R Young. 1988 · 1988
Earlier work this paper cites.
Empirical methods for artificial intelligence . Vol. 139
Paul R Cohen. 1995 · 1995
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998 · 1998
Earlier work this paper cites.
Random forests
Leo Breiman. 2001 · 2001
Earlier work this paper cites.
Why most published research findings are false
John PA Ioannidis. 2005 · 2005
Earlier work this paper cites.
Replicability is not reproducibility: nor is it good science. In Proc. of the Evaluation Methods for Machine Learning Workshop at the 26th International Conference on Machine Learning, Montreal, Canada
Chris Drummond. 2009 · 2009
Earlier work this paper cites.
Improving numerical reproducibility and stability in large-scale numerical simulations on GPUs. In 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS) . IEEE, 1–9
Michela Taufer, Omar Padron, Philip Saponaro, and Sandeep Patel. 2010 · 2010
Earlier work this paper cites.
Reproducible research in computational science
Roger D. Peng. 2011 · 2011
Earlier work this paper cites.
Unbiased look at dataset bias. In CVPR 2011 . IEEE, 1521–1528
Antonio Torralba and Alexei A Efros. 2011 · 2011
Earlier work this paper cites.
International vocabulary of metrology – Basic and general concepts and associated terms - 3rd edition with minor corrections
Joint Committee for Guides in Metrology. 2012 · 2012
Earlier work this paper cites.
Power failure: why small sample size undermines the reliability of neuroscience
Katherine S Button, John Ioannidis, Claire Mokrysz, Brian A Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R Munafò. 2013 · 2013
Earlier work this paper cites.
A cross disciplinary study of link decay and the effectiveness of mitigation techniques. In BMC bioinformatics , Vol. 14. BioMed Central, 1–11
Jason Hennessey and Steven Xijin Ge. 2013 · 2013
Earlier work this paper cites.
An evaluation of the software system dependency of a global atmospheric model
Song-You Hong, Myung-Seo Koo, Jihyeon Jang, Jung-Eun Esther Kim, Hoon Park, Min-Su Joh, Ji-Hoon Kang, and Tae-Jin Oh. 2013 · 2013
Earlier work this paper cites.
P-curve: a key to the file-drawer
Uri Simonsohn, Leif D Nelson, and Joseph P Simmons. 2014 · 2014
Earlier work this paper cites.
Computer Scientists Are Astir After Baidu Team Is Barred From A.I. Competition
John Markoff. 2015 · 2015
Earlier work this paper cites.
Estimating the reproducibility of psychological science
Open Science Collaboration. 2015 · 2015
Earlier work this paper cites.
Reproducing statistical results
Victoria Stodden. 2015 · 2015
Earlier work this paper cites.
Reproducibility crisis
Monya Baker. 2016 · 2016
Earlier work this paper cites.
What does research reproducibility mean?
Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. 2016 · 2016
Earlier work this paper cites.
Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 338–348
Nils Reimers and Iryna Gurevych. 2017 · 2017
Earlier work this paper cites.
Questionable answers in question answering research: Reproducibility and variability of published results
Matt Crane. 2018 · 2018
Earlier work this paper cites.
State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32
Odd Erik Gundersen and Sigbjørn Kjensmo. 2018 · 2018
Earlier work this paper cites.
Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018 · 2018
Earlier work this paper cites.
Artificial intelligence faces reproducibility crisis
Matthew Hutson. 2018 · 2018
Earlier work this paper cites.
The gan landscape: Losses, architectures, regularization, and normalization. In ICML 2018 Workshop on Reproducibility in Machine Learning
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. 2018 · 2018
Earlier work this paper cites.
Are GANs Created Equal? A Large-Scale Study. In NeurIPS
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. 2018 · 2018
Earlier work this paper cites.
Statistical and Machine Learning forecasting methods: Concerns and ways forward
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2018 · 2018
Earlier work this paper cites.
On the State of the Art of Evaluation in Neural Language Models. In International Conference on Learning Representations
Gábor Melis, Chris Dyer, and Phil Blunsom. 2018 · 2018
Cited alongside, same era.
Statistics and chemometrics for analytical chemistry
James Miller and Jane C Miller. 2018 · 2018
Cited alongside, same era.
Reproducibility vs. replicability: a brief history of a confused terminology
Hans E Plesser. 2018 · 2018
Cited alongside, same era.
Winner’s curse? On pace, progress, and empirical rigor. In ICLR 2018 Workshop Track
David Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. 2018 · 2018
Cited alongside, same era.
Unreproducible research is reproducible. In International Conference on Machine Learning . PMLR, 725–734
Xavier Bouthillier, César Laurent, and Pascal Vincent. 2019 · 2019
Cited alongside, same era.
Accounting for variance in machine learning benchmarks
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al · 2021
Later among the works it cites.
Hyperparameter Optimization Is Deceiving Us, and How to Stop It
A Feder Cooper, Yucheng Lu, Jessica Forde, and Christopher M De Sa. 2021 · 2021
Later among the works it cites.
Progress in recommender systems research: Crisis? What crisis?
Paolo Cremonesi and Dietmar Jannach. 2021 · 2021
Later among the works it cites.
A troubling analysis of reproducibility and progress in recommender systems research
Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2021 · 2021
Later among the works it cites.
Complications for Computational Experiments from Modern Processors. In 27th International Conference on Principles and Practice of Constraint Programming (CP 2021) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik
Johannes K Fichte, Markus Hecher, Ciaran McCreesh, and Anas Shahab. 2021 · 2021
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems . 101–109
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019 · 2019
Cited alongside, same era.
Show Your Work: Improved Reporting of Experimental Results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 2185–2194
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. 2019 · 2019
Cited alongside, same era.
Standing on the Feet of Giants - Reproducibility in AI
Odd Erik Gundersen. 2019 · 2019
Cited alongside, same era.
Troubling Trends in Machine Learning Scholarship: Some ML Papers Suffer from Flaws That Could Mislead the Public and Stymie Future Research
Zachary C. Lipton and Jacob Steinhardt. 2019 · 2019
Cited alongside, same era.
The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. 2019 · 2019
Cited alongside, same era.
Reproducibility and replicability in science
Engineering National Academies of Sciences, Medicine, et al · 2019
Cited alongside, same era.
A step toward quantifying independently reproducible machine learning research
Edward Raff. 2019 · 2019
Cited alongside, same era.
Later among the works it cites.
The clinician and dataset shift in artificial intelligence
Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane, and Suchi Saria. 2021 · 2021
Later among the works it cites.
Datasheets for datasets
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021 · 2021
Later among the works it cites.
Replication across space and time must be weak in the social and environmental sciences
Michael F Goodchild and Wenwen Li. 2021 · 2021
Later among the works it cites.
The Case Against Registered Reports
Odd Erik Gundersen. 2021a · 2021
Later among the works it cites.
The fundamental principles of reproducibility
Odd Erik Gundersen. 2021b · 2021
Later among the works it cites.
Session-aware recommendation: A surprising quest for the state-of-the-art
Sara Latifi, Noemi Mauro, and Dietmar Jannach. 2021 · 2021
Later among the works it cites.
Reproducibility in machine learning for health research: Still a ways to go
Matthew BA McDermott, Shirly Wang, Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. 2021 · 2021
Later among the works it cites.
Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021 · 2021
Later among the works it cites.
Wagner Gonçalves Pinto, Antonio Alguacil, and Michaël Bauerheim. 2021 · 2021
Later among the works it cites.
Research Reproducibility as a Survival Analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 469–478
Edward Raff. 2021 · 2021
Later among the works it cites.
Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, et al · 2021
Later among the works it cites.
Randomness in neural network training: Characterizing the impact of tooling
Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. 2021 · 2021
Later among the works it cites.
Could machine learning fuel a reproducibility crisis in science?
Elizabeth Gibney. 2022 · 2022
Closest in time.
Do machine learning platforms provide out-of-the-box reproducibility?
Odd Erik Gundersen, Saeid Shamsaliei, and Richard Juul Isdahl. 2022 · 2022
Closest in time.
The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society . 335–348
Jessica Hullman, Sayash Kapoor, Priyanka Nanayakkara, Andrew Gelman, and Arvind Narayanan. 2022 · 2022
Closest in time.
Leakage and the Reproducibility Crisis in ML-based Science
Sayash Kapoor and Arvind Narayanan. 2022 · 2022
Closest in time.
The fallacy of AI functionality. In 2022 ACM Conference on Fairness, Accountability, and Transparency . 959–972
Inioluwa Deborah Raji, I Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022 · 2022
Closest in time.
How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models?
Mostafa Shahriari, Rudolf Ramler, and Lukas Fischer. 2022 · 2022
Closest in time.
Machine learning for medical imaging: methodological failures and recommendations for the future
Gaël Varoquaux and Veronika Cheplygina. 2022 · 2022
Closest in time.
A Thorough Reproducibility Study on Sentiment Classification: Methodology, Experimental Setting, Results
Giorgio Maria Di Nunzio and Riccardo Minzoni. 2023 · 2023
Closest in time.
Improving Reproducibility in AI Research: Four Mechanisms Adopted by JAIR
Odd Erik Gundersen, Malte Helmert, and Holger Hoos. 2023 · 2023
Closest in time.