Fetching the paper…
Reading the bibliography…
Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so far emerged.
Biometry: The Principles and Practice of Statistics in Biological Research
R.R. Sokal and F.J. Rohlf. 1971 · 1971
Earlier work this paper cites.
Linear statistical inference and its applications
Calyampudi Radhakrishna Rao. 1973 · 1973
Earlier work this paper cites.
A pooling methodology for coefficient of variation
SE Ahmed. 1995 · 1995
Earlier work this paper cites.
The need for open source software in machine learning
Soren Sonnenburg, Mikio L Braun, Cheng Soon Ong, Samy Bengio, Leon Bottou, Geoffrey Holmes, Yann LeCunn, Klaus-Robert Muller, Fernando Pereira, Carl Edward Rasmussen, et al. 2007 · 2007
Earlier work this paper cites.
Empiricism is not a matter of faith
Ted Pedersen. 2008 · 2008
Earlier work this paper cites.
Replicability is not reproducibility: nor is it good science
Chris Drummond. 2009 · 2009
Earlier work this paper cites.
International vocabulary of metrology-basic and general concepts and associated terms (VIM)
JCGM. 2012 · 2012
Earlier work this paper cites.
Reproducibility crisis
Monya Baker. 2016 · 2016
Earlier work this paper cites.
Sustainable computational science: the rescience initiative
Nicolas P Rougier, Konrad Hinsen, Frédéric Alexandre, Thomas Arildsen, Lorena A Barba, Fabien CY Benureau, C Titus Brown, Pierre De Buyl, Ozan Caglayan, Andrew P Davison, et al. 2017 · 2017
Earlier work this paper cites.
Pass: A dutch data-to-text system for soccer, targeted towards specific audiences
Chris van der Lee, Emiel Krahmer, and Sander Wubben. 2017 · 2017
Cited alongside, same era.
The MT Reproducibility Checklist
Kirstie Whitaker. 2017 · 2017
Cited alongside, same era.
Identifying and overcoming threats to reproducibility, replicability, robustness, and generalizability in microbiome research
Patrick D Schloss. 2018 · 2018
Cited alongside, same era.
Experiments with universal CEFR classification
Sowmya Vajjala and Taraka Rama. 2018 · 2018
Cited alongside, same era.
Reproducibility in computational linguistics: Are we willing to share?
Martijn Wieling, Josine Rawee, and Gertjan van Noord. 2018 · 2018
Cited alongside, same era.
Community perspective on replicability in natural language processing
A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with REPROLANG2020
António Branco, Nicoletta Calzolari, Piek Vossen, Gertjan Van Noord, Dieter van Uytvanck, João Silva, Luís Gomes, André Moreira, and Willem Elbers. 2020 · 2020
Later among the works it cites.
REPROLANG 2020: Automatic proficiency scoring of Czech, English, German, Italian, and Spanish learner essays
Andrew Caines and Paula Buttery. 2020 · 2020
Later among the works it cites.
Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions
David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020 · 2020
Later among the works it cites.
Reproduction and replication: A case study with automatic essay scoring
Eva Huber and Çağrı Çöltekin. 2020 · 2020
Later among the works it cites.
The machine learning reproducibility checklist v2.0
Joelle Pineau. 2020 · 2020
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Margot Mieskes, Karën Fort, Aurélie Névéol, Cyril Grouin, and Kevin Cohen. 2019 · 2019
Cited alongside, same era.
Language proficiency scoring
Cristina Arhiliuc, Jelena Mitrović, and Michael Granitzer. 2020 · 2020
Cited alongside, same era.
Artifact review and badging Version 1.1, August 24, 2020
Association for Computing Machinery. 2020 · 2020
Cited alongside, same era.
Reproducing monolingual, multilingual and cross-lingual CEFR predictions
Yves Bestgen. 2020 · 2020
Cited alongside, same era.
Assay validation methods: Definitions and terms
AAH/USFWS. n.d
Cited in the paper.
ReproGen: Proposal for a shared task on reproducibility of human evaluations in NLG
Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2020a
Cited in the paper.
Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing
Anya Belz, Simon Mille, and David M. Howcroft. 2020b
Cited in the paper.
Later among the works it cites.
A systematic review of reproducibility research in natural language processing
Anja Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021 · 2021
Closest in time.
Another pass: A reproduction study of the human evaluation of a football report generation system
Simon Mille, Thiago Castro Ferreira, Anya Belz, and Brian Davis. 2021 · 2021
Closest in time.
Anastasia Shimorina and Anya Belz. 2021 · 2021
Closest in time.