Paper

MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning

Rajan, Raghu, Diaz, Jessica Lizeth Borja, Guttikonda, Suresh et al.

Understand

We present MDP Playground, a testbed for Reinforcement Learning (RL) agents with dimensions of hardness that can be controlled independently to challenge agents in different ways and obtain varying degrees of hardness in toy and complex RL environments.

We consider and allow control over a wide variety of dimensions, including delayed rewards, sequence lengths, reward density, stochasticity, image representations, irrelevant features, time unit, action range and more.
We define a parameterised collection of fast-to-run toy environments in OpenAI Gym by varying these dimensions and propose to use these to understand agents better.
We then show how to design experiments using MDP Playground to gain insights on the toy environments.

Built on

Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments
Young, K., and Tian, T. (2019) · 1903
Earlier work this paper cites.
Toybox: A suite of environments for experimental evaluation of deep reinforcement learning
Tosch, E., Clary, K., Foley, J., and Jensen, D. D. (2019) · 1905
Earlier work this paper cites.
MNIST-C: A robustness benchmark for computer vision
Mu, N., and Gilmer, J. (2019) · 1906
Earlier work this paper cites.
Benchmarking model-based reinforcement learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J. (2019) · 1907
Earlier work this paper cites.
Quantifying the carbon emissions of machine learning
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019) · 1910
Earlier work this paper cites.
Asynchronous methods for deep reinforcement learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016) · 1937
Earlier work this paper cites.
Intuitive choices lead to intensified positive emotions: An overlooked reason for “intuition bias”?
Kirkebøen, G., and Nordbye, G. H. (2017) · 1942
Earlier work this paper cites.
Pathology on game trees revisited, and an alternative to minimaxing
Nau, D. S. (1983) · 1983
Earlier work this paper cites.
Acting optimally in partially observable stochastic domains
Cassandra, A. R., Kaelbling, L. P., and Littman, M. L. (1994) · 1994
Earlier work this paper cites.
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L. (1994) · 1994
Earlier work this paper cites.
Rewarding behaviors
Bacchus, F., Boutilier, C., and Grove, A. (1996) · 1996
Earlier work this paper cites.
Reinforcement learning - an introduction
Sutton, R. S., and Barto, A. G. (1998) · 1998
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., and Singh, S. (1999) · 1999
Earlier work this paper cites.
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2020) · 2003
Earlier work this paper cites.
Autonomous helicopter flight via reinforcement learning
Kim, H. J., Jordan, M. I., Sastry, S., and Ng, A. Y. (2004) · 2004
Earlier work this paper cites.
Probabilistic robotics
Thrun, S., Burgard, W., and Fox, D. (2005) · 2005
Earlier work this paper cites.
Near-optimal regret bounds for reinforcement learning
Jaksch, T., Ortner, R., and Auer, P. (2010) · 2010
Earlier work this paper cites.
MNIST handwritten digit database
LeCun, Y., and Cortes, C. (2010) · 2010
Earlier work this paper cites.
On adversarial search spaces and sampling-based planning
Ramanujan, R., Sabharwal, A., and Selman, B. (2010) · 2010
Earlier work this paper cites.
Approximate information state for approximate planning and reinforcement learning in partially observed systems
Subramanian, J., Sinha, A., Seraj, R., and Mahajan, A. (2020) · 2010
Earlier work this paper cites.
Double q-learning
van Hasselt, H. (2010) · 2010
Earlier work this paper cites.

Similar

Learning invariant feature hierarchies
LeCun, Y. (2012) · 2012
Cited alongside, same era.
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y. (2012) · 2012
Cited alongside, same era.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013) · 2013
Cited alongside, same era.
How hard is my mdp?" the distribution-norm to the rescue"
Maillard, O., Mann, T. A., and Mannor, S. (2014) · 2014
Cited alongside, same era.
How to discount deep reinforcement learning: Towards new dynamic strategies.
François-Lavet, V., Fonteneau, R., and Ernst, D. (2015) · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015) · 2015
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016) · 2016
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C., Abbeel, P., and Levine, S. (2017) · 2017
Cited alongside, same era.
Environment-independent task specifications via GLTL
Littman, M. L., Topcu, U., Fu, J., Jr., C. L. I., Wen, M., and MacGlashan, J. (2017) · 2017
Cited alongside, same era.
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017) · 2017
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017) · 2017
Cited alongside, same era.
Toward robust deep rl via better benchmarks: Identifying neglected problem dimensions
Andersson, O., and Doherty, P. (2018) · 2018
Cited alongside, same era.

Then

Learning end-to-end goal-oriented dialog with multiple answers
Rajendran, J., Ganhotra, J., Singh, S., and Polymenakos, L. (2018) · 2018
Later among the works it cites.
Meta-gradient reinforcement learning
Xu, Z., van Hasselt, H. P., and Silver, D. (2018) · 2018
Later among the works it cites.
On learning intrinsic rewards for policy gradient methods
Zheng, Z., Oh, J., and Singh, S. (2018) · 2018
Later among the works it cites.
RUDDER: return decomposition for delayed rewards
Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., and Hochreiter, S. (2019) · 2019
Later among the works it cites.
Tackling sparse rewards in real-time games with statistical forward planning methods
Gaina, R. D., Lucas, S. M., and Pérez-Liébana, D. (2019) · 2019
Later among the works it cites.
Benchmarking neural network robustness to common corruptions and perturbations
Hendrycks, D., and Dietterich, T. G. (2019) · 2019
Later among the works it cites.
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning
Juliani, A., Khalifa, A., Berges, V., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D. (2019) · 2019
Later among the works it cites.
Self-paced contextual reinforcement learning
Klink, P., Abdulsamad, H., Belousov, B., and Peters, J. (2019) · 2019
Later among the works it cites.
Behaviour suite for reinforcement learning
Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepezvari, C., Singh, S., Roy, B. V., Sutton, R., Silver, D., and Hasselt, H. V. (2019) · 2019
Later among the works it cites.
Learning to design rna
Runge, F., Stoll, D., Falkner, S., and Hutter, F. (2019) · 2019
Later among the works it cites.
Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning
Atrey, A., Clary, K., and Jensen, D. D. (2020) · 2020
Later among the works it cites.
Implementation matters in deep RL: A case study on PPO and TRPO
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. (2020) · 2020
Later among the works it cites.
Learning to utilize shaping rewards: A new approach of reward shaping
Hu, Y., Wang, W., Jia, H., Wang, Y., Chen, Y., Hao, J., Wu, F., and Fan, C. (2020) · 2020
Later among the works it cites.
Action space shaping in deep reinforcement learning
Kanervisto, A., Scheller, C., and Hautamäki, V. (2020) · 2020
Later among the works it cites.
Reward tweaking: Maximizing the total reward while planning for short horizons.
Tessler, C., and Mannor, S. (2020) · 2020
Later among the works it cites.
What matters for on-policy deep actor-critic methods? A large-scale study
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O. (2021) · 2021
Later among the works it cites.
TempoRL: Learning when to act
Biedenkapp, A., Rajan, R., Hutter, F., and Lindauer, M. (2021) · 2021
Later among the works it cites.
The benchmark lottery
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., and Vinyals, O. (2021) · 2021
Later among the works it cites.
On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., Hutter, F., and Calandra, R. (2021) · 2021
Later among the works it cites.
Learning task-distribution reward shaping with meta-learning
Zou, H., Ren, T., Yan, D., Su, H., and Zhu, J. (2021) · 2021
Later among the works it cites.
Leveraging procedural generation to benchmark reinforcement learning
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. (2020) · 2056
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…