How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Built on

Welch, B. L., 1947. The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika 34 (1/2), 28–35
1947
Earlier work this paper cites.
Rice, W. R., 1989. Analyzing tables of statistical tests. Evolution 43 (1), 223–225
1989
Earlier work this paper cites.
Cabin, R. J., Mitchell, R. J., 2000. To bonferroni or not to bonferroni: when and how are the questions. Bulletin of the Ecological Society of America 81 (3), 246–248
2000
Earlier work this paper cites.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D., 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971
Original
2015
Earlier work this paper cites.
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P., 2015. Trust region policy optimization. CoRR, abs/1502.05477
Original
2015
Earlier work this paper cites.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W., 2016. Openai gym. arXiv preprint arXiv:1606.01540
Original
2016
Cited alongside, same era.
Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., 2017. Openai baselines. https://github.com/openai/baselines
2017
Cited alongside, same era.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D., 2017. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560
Original
2017
Cited alongside, same era.
Islam, R., Henderson, P., Gomrokchi, M., Precup, D., 2017. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. In: Proceedings of the ICML 2017 workshop on Reproducibility in Machine Learning (RML)
2017
Cited alongside, same era.