A Step Toward Quantifying Independently Reproducible Machine Learning Research

F. Yates, “Contingency Tables Involving Small Numbers and the χ \chi 2 Test,” Supplement to the Journal of the Royal Statistical Society , vol. 1, no. 2, pp. 217–235, 1934. [Online]. Available: http://www.jstor.org/stable/2983604

1934

Earlier work this paper cites.

H. B. Mann and D. R. Whitney, “On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other,” The Annals of Mathematical Statistics , vol. 18, no. 1, pp. 50–60, 3 1947. [Online]. Available: http://projecteuclid.org/euclid.aoms/1177730491

1947

Earlier work this paper cites.

W. H. Kruskal and W. A. Wallis, “Use of Ranks in One-Criterion Variance Analysis,” Journal of the American Statistical Association , vol. 47, no. 260, pp. 583–621, 12 1952. [Online]. Available: https://www.tandfonline.com/doi/abs/10.1080/01621459.1952.10483441

1952

Earlier work this paper cites.

O. J. Dunn, “Multiple Comparisons Among Means,” Journal of the American Statistical Association , vol. 56, no. 293, pp. 52–64, 1961. [Online]. Available: http://www.jstor.org/stable/2282330

1961

Earlier work this paper cites.

S. S. Shapiro and M. B. Wilk, “An analysis of variance test for normality (complete samples),” Biometrika , vol. 52, no. 3-4, pp. 591–611, 12 1965. [Online]. Available: https://academic.oup.com/biomet/article-lookup/doi/10.1093/biomet/52.3-4.591

1965

Earlier work this paper cites.

C. Drummond, “Replicability is not reproducibility: nor is it good science,” in Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, Montreal, Canada,2009 , ser. Evaluation Methods for Machine Learning Workshop, the 26th ICML, June 14-18, 2009, Montreal, Canada, 2009

2009

Earlier work this paper cites.

R. Dror, G. Baumer, M. Bogomolov, and R. Reichart, “Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets,” Transactions of the Association for Computational Linguistics , vol. 5, pp. 471–486, 12 2017. [Online]. Available: https://www.aclweb.org/anthology/Q17-1033

2017

Cited alongside, same era.

A. Rahimi and B. Recht, “NIPS 2017 Test-of-time award presentation,” 2017

2017

Cited alongside, same era.

E. Raff, “JSAT: Java Statistical Analysis Tool, a Library for Machine Learning,” Journal of Machine Learning Research , vol. 18, no. 23, pp. 1–5, 2017. [Online]. Available: http://jmlr.org/papers/v18/16-131.html

2017

Cited alongside, same era.

M. Hutson, “Artificial intelligence faces reproducibility crisis,” Science , vol. 359, no. 6377, pp. 725–726, 2018. [Online]. Available: https://science.sciencemag.org/content/359/6377/725

2018

Cited alongside, same era.

G. C. Publio, D. Esteves, and H. Zafar, “ML-Schema : Exposing the Semantics of Machine Learning with Schemas and Ontologies,” in Reproducibility in ML Workshop, ICML’18 , 2018

2018

Later among the works it cites.

J. Forde, T. Head, C. Holdgraf, Y. Panda, F. Perez, G. Nalvarte, B. Ragan-kelley, and E. Sundell, “Reproducible Research Environments with repo2docker,” in Reproducibility in ML Workshop, ICML’18 , 2018

2018

Later among the works it cites.

O. E. Gundersen and S. Kjensmo, “State of the Art: Reproducibility in Artificial Intelligence,” Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18) , pp. 1644–1651, 2018

2018

Later among the works it cites.

D. Sculley, J. Snoek, A. Rahimi, and A. Wiltschko, “Winner’s Curse? On Pace, Progress, and Empirical Rigor,” in ICLR Workshop track , 2018. [Online]. Available: https://openreview.net/pdf?id=rJWF0Fywf

2018

Later among the works it cites.