A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

Student, “The probable error of a mean,” Biometrika , pp. 1–25, 1908

1908

Earlier work this paper cites.

C. Bonferroni, “Teoria statistica delle classi e calcolo delle probabilita,” Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze , vol. 8, pp. 3–62, 1936

Original

1936

Earlier work this paper cites.

F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics bulletin , vol. 1, no. 6, pp. 80–83, 1945

1945

Earlier work this paper cites.

B. L. Welch, “The generalization of student’s’ problem when several different population variances are involved,” Biometrika , vol. 34, no. 1/2, pp. 28–35, 1947

1947

Earlier work this paper cites.

N. Smirnov, “Table for estimating the goodness of fit of empirical distributions,” The annals of mathematical statistics , vol. 19, no. 2, pp. 279–281, 1948

1948

Earlier work this paper cites.

W. J. Conover and R. L. Iman, “Rank transformations as a bridge between parametric and nonparametric statistics,” The American Statistician , vol. 35, no. 3, pp. 124–129, 1981

1981

Cited alongside, same era.

B. Efron and R. J. Tibshirani, An introduction to the bootstrap . CRC press, 1994

1994

Cited alongside, same era.

G. K. Kanji, 100 statistical tests . Sage, 2006

2006

Cited alongside, same era.

J. C. De Winter, “Using the student’s t-test with extremely small sample sizes.” Practical Assessment, Research & Evaluation , vol. 18, no. 10, 2013

2013

Cited alongside, same era.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540 , 2016

Original

2016

Cited alongside, same era.

R. Islam, P. Henderson, M. Gomrokchi, and D. Precup, “Reproducibility of benchmarked deep reinforcement learning tasks for continuous control,” in Proceedings of the ICML 2017 workshop on Reproducibility in Machine Learning (RML) , 2017

2017

Later among the works it cites.

P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” arXiv preprint arXiv:1709.06560 , 2017

Original

2017

Later among the works it cites.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” arXiv preprint arXiv:1801.01290 , 2018

Original

2018

Later among the works it cites.

S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” arXiv preprint arXiv:1802.09477 , 2018

Original

2018

Later among the works it cites.