2022

Evolving Curricula with Regret-Based Environment Design

Parker-Holder, Jack, Jiang, Minqi, Dennis, Michael et al.

Understand

It remains a significant challenge to train generally capable agents with reinforcement learning (RL).

A promising avenue for improving the robustness of RL agents is through the use of curricula.
One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities.
These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces.

Built on

Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions
Original
Wang, R., Lehman, J., Clune, J., and Stanley, K. O. (2019) · 1901
Earlier work this paper cites.
Solving rubik’s cube with a robot hand
Original
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. (2019) · 1910
Earlier work this paper cites.
Dota 2 with large scale deep reinforcement learning
Original
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S. (2019) · 1912
Earlier work this paper cites.
The theory of statistical decision
Savage, L. J. (1951) · 1951
Earlier work this paper cites.
Biological extinction in earth history
Raup, D. M. (1986) · 1986
Earlier work this paper cites.
Evolutionary robotics and the radical envelope-of-noise hypothesis
Jakobi, N. (1997) · 1997
Earlier work this paper cites.
Introduction to Reinforcement Learning
Sutton, R. S. and Barto, A. G. (1998) · 1998
Earlier work this paper cites.
Generating automatic curricula via self-supervised active domain randomization
Original
Raparthy, S. C., Mehta, B., Golemo, F., and Paull, L. (2020) · 2002
Earlier work this paper cites.
Automatic curriculum learning for deep rl: A short survey
Original
Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P.-Y. (2020) · 2003
Earlier work this paper cites.
Motor skill acquisition under environmental perturbations: On the necessity of alternate freezing and freeing of degrees of freedom
Berthouze, L. and Lungarella, M. (2004) · 2004
Earlier work this paper cites.
An experiment in automatic game design
Togelius, J. and Schmidhuber, J. (2008) · 2008
Earlier work this paper cites.
A general approach to environment design with one agent
Zhang, H., Chen, Y., and Parkes, D. (2009) · 2008
Earlier work this paper cites.
Value-based policy teaching with active indirect elicitation
Zhang, H. and Parkes, D. (2008) · 2008
Earlier work this paper cites.
Robust intrinsically motivated exploration and active learning
Baranes, A. and Oudeyer, P.-Y. (2009) · 2009
Earlier work this paper cites.
Generalized domains for empirical evaluations in reinforcement learning
Whiteson, S., Tanner, B., Taylor, M. E., and Stone, P. (2009) · 2009
Earlier work this paper cites.
Automatic curriculum learning through value disagreement
Zhang, Y., Abbeel, P., and Pinto, L. (2020) · 2009
Earlier work this paper cites.
Playing atari with deep reinforcement learning
Original
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. (2013) · 2013
Earlier work this paper cites.
Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem
Schmidhuber, J. (2013) · 2013
Earlier work this paper cites.
Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria
Soros, L. and Stanley, K. (2014) · 2014
Earlier work this paper cites.
Extinction events can accelerate evolution
Lehman, J. and Miikkulainen, R. (2015) · 2015
Earlier work this paper cites.
OpenAI Gym
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016) · 2016
Earlier work this paper cites.
Quality diversity: A new frontier for evolutionary computation
Pugh, J. K., Soros, L. B., and Stanley, K. O. (2016) · 2016
Earlier work this paper cites.
High-dimensional continuous control using generalized advantage estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016) · 2016
Earlier work this paper cites.
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T. P., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. (2016) · 2016
Earlier work this paper cites.
Hindsight experience replay
Andrychowicz, M., Crow, D., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2017) · 2017
Earlier work this paper cites.
Minimal criterion coevolution: A new approach to open-ended search
Brant, J. C. and Stanley, K. O. (2017) · 2017
Earlier work this paper cites.
Reverse curriculum generation for reinforcement learning
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. (2017) · 2017
Earlier work this paper cites.
Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task
James, S., Davison, A. J., and Johns, E. (2017) · 2017
Earlier work this paper cites.
Equi-reward utility maximizing design in stochastic environments
Keren, S., Pineda, L., Gal, A., Karpas, E., and Zilberstein, S. (2017) · 2017
Earlier work this paper cites.
Markov decision processes with continuous side information
Modi, A., Jiang, N., Singh, S., and Tewari, A. (2017) · 2017
Earlier work this paper cites.

Similar

Sim-to-real transfer of robotic control with dynamics randomization
Original
Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2017) · 2017
Cited alongside, same era.
Supervision via competition: Robot adversaries for learning tasks
Pinto, L., Davidson, J., and Gupta, A. (2017) · 2017
Cited alongside, same era.
CAD2RL: real single-image flight without a single real image
Sadeghi, F. and Levine, S. (2017) · 2017
Cited alongside, same era.
Open-endedness: The last grand challenge you’ve never heard of
Stanley, K. O., Lehman, J., and Soros, L. (2017) · 2017
Cited alongside, same era.
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017) · 2017
Cited alongside, same era.
Minimalistic gridworld environment for OpenAI Gym
Chevalier-Boisvert, M., Willems, L., and Pal, S. (2018) · 2018
Cited alongside, same era.
Automatic goal generation for reinforcement learning agents
Florensa, C., Held, D., Geng, X., and Abbeel, P. (2018) · 2018
Cited alongside, same era.
Procedural level generation improves generality of deep reinforcement learning
Original
Justesen, N., Torrado, R. R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S. (2018) · 2018
Cited alongside, same era.
Pytorch implementations of reinforcement learning algorithms
Kostrikov, I. (2018) · 2018
Cited alongside, same era.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2016-2018) · 2018
Cited alongside, same era.
Intrinsic motivation and automatic curricula via asymmetric self-play
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. (2018) · 2018
Cited alongside, same era.
Procedural content generation via machine learning (PCGML)
Summerville, A., Snodgrass, S., Guzdial, M., Holmgård, C., Hoover, A. K., Isaksen, A., Nealen, A., and Togelius, J. (2018) · 2018
Cited alongside, same era.

Then

The NetHack Learning Environment
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. (2020) · 2020
Later among the works it cites.
Reinforcement learming with augmented data
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. (2020) · 2020
Later among the works it cites.
Teacher-student curriculum learning
Matiisen, T., Oliver, A., Cohen, T., and Schulman, J. (2020) · 2020
Later among the works it cites.
On gradient-based learning in continuous games
Mazumdar, E., Ratliff, L. J., and Sastry, S. S. (2020) · 2020
Later among the works it cites.
Active domain randomization
Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, L. (2020) · 2020
Later among the works it cites.
Skew-fit: State-covering self-supervised reinforcement learning
Pong, V., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. (2020) · 2020
Later among the works it cites.
Automated curriculum generation through setter-solver interactions
Racaniere, S., Lampinen, A., Santoro, A., Reichert, D., Firoiu, V., and Lillicrap, T. (2020) · 2020
Later among the works it cites.
Increasing generality in machine learning through procedural content generation
Risi, S. and Togelius, J. (2020) · 2020
Later among the works it cites.
Observational overfitting in reinforcement learning
Song, X., Jiang, Y., Tu, S., Du, Y., and Neyshabur, B. (2020) · 2020
Later among the works it cites.
The unexpected consequence of incremental design changes
Sturtevant, N., Decroocq, N., Tripodi, A., and Guzdial, M. (2020) · 2020
Later among the works it cites.
Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions
Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. (2020) · 2020
Later among the works it cites.
Augmented world models facilitate zero-shot dynamics generalization from a single offline environment
Ball, P. J., Lu, C., Parker-Holder, J., and Roberts, S. J. (2021) · 2021
Later among the works it cites.
Learning with AMIGo: Adversarially motivated intrinsic goals
Campero, A., Raileanu, R., Kuttler, H., Tenenbaum, J. B., Rocktäschel, T., and Grefenstette, E. (2021) · 2021
Later among the works it cites.
Self-paced context evaluation for contextual reinforcement learning
Eimer, T., Biedenkapp, A., Hutter, F., and Lindauer, M. (2021) · 2021
Later among the works it cites.
Adaptive procedural task generation for hard-exploration problems
Fang, K., Zhu, Y., Savarese, S., and Li, F.-F. (2021) · 2021
Later among the works it cites.
On the importance of environments in human-robot coordination
Fontaine, M. C., Hsu, Y., Zhang, Y., Tjanaka, B., and Nikolaidis, S. (2021) · 2021
Later among the works it cites.
Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability
Original
Ghosh, D., Rahme, J., Kumar, A., Zhang, A., Adams, R. P., and Levine, S. (2021) · 2021
Later among the works it cites.
Adversarial environment generation for learning to navigate the web
Gur, I., Jaques, N., Malta, K., Tiwari, M., Lee, H., and Faust, A. (2021) · 2021
Later among the works it cites.
A survey of generalisation in deep reinforcement learning
Original
Kirk, R., Zhang, A., Grefenstette, E., and Rocktäschel, T. (2021) · 2021
Later among the works it cites.
Image augmentation is all you need: Regularizing deep reinforcement learning from pixels
Kostrikov, I., Yarats, D., and Fergus, R. (2021) · 2021
Later among the works it cites.
Deep learning for procedural content generation
Liu, J., Snodgrass, S., Khalifa, A., Risi, S., Yannakakis, G. N., and Togelius, J. (2021) · 2021
Later among the works it cites.
Discovering and achieving goals via world models
Mendonca, R., Rybkin, O., Daniilidis, K., Hafner, D., and Pathak, D. (2021) · 2021
Later among the works it cites.
Asymmetric self-play for automatic goal discovery in robotic manipulation
OpenAI, O., Plappert, M., Sampedro, R., Xu, T., Akkaya, I., Kosaraju, V., Welinder, P., D’Sa, R., Petron, A., de Oliveira Pinto, H. P., Paino, A., Noh, H., Weng, L., Yuan, Q., Chu, C., and Zaremba, W. (2021) · 2021
Later among the works it cites.
Automatic data augmentation for generalization in deep reinforcement learning
Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., and Fergus, R. (2021) · 2021
Later among the works it cites.
Minihack the planet: A sandbox for open-ended reinforcement learning research
Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E., Petroni, F., Kuttler, H., Grefenstette, E., and Rocktäschel, T. (2021) · 2021
Later among the works it cites.
Reward is enough
Silver, D., Singh, S., Precup, D., and Sutton, R. S. (2021) · 2021
Later among the works it cites.
Open-ended learning leads to generally capable agents
Original
Team, O. E. L., Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J., Trebacz, M., Jaderberg, M., Mathieu, M., McAleese, N., Bradley-Schmieg, N., Wong, N., Porcel, N., Raileanu, R., Hughes-Fitt, S., Dalibard, V., and Czarnecki, W. M. (2021) · 2021
Later among the works it cites.
Automated reinforcement learning (autorl): A survey and open problems
Original
Parker-Holder, J., Rajan, R., Song, X., Biedenkapp, A., Miao, Y., Eimer, T., Zhang, B., Nguyen, V., Calandra, R., Faust, A., Hutter, F., and Lindauer, M. (2022) · 2022
Closest in time.
Leveraging procedural generation to benchmark reinforcement learning
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. (2020) · 2056
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…