2020

Automatic Curriculum Learning For Deep RL: A Short Survey

Portelas, Rémy, Colas, Cédric, Weng, Lilian et al.

Understand

Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL).These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities.

In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others.
The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.

Built on

Samples are not all useful: Denoising policy gradient updates using variance
Original
Yannis Flet-Berliac and Philippe Preux · 1904
Earlier work this paper cites.
Alan M Turing
1950
Earlier work this paper cites.
Training and tracking in robotics
Oliver G Selfridge et al · 1985
Earlier work this paper cites.
Curious model-building control systems
Jürgen Schmidhuber · 1991
Earlier work this paper cites.
Self-improving reactive agents based on reinforcement learning, planning and teaching
Long-Ji Lin · 1992
Earlier work this paper cites.
Learning and development in neural networks: the importance of starting small
Jeffrey L. Elman · 1993
Earlier work this paper cites.
Prioritized sweeping: Reinforcement learning with less data and less time
Andrew W Moore and Christopher G Atkeson · 1993
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Pierre-Yves Oudeyer et al · 2007
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio et al · 2009
Earlier work this paper cites.
Flexible shaping: How learning in small steps helps
Kai A. Krueger and Peter Dayan · 2009
Earlier work this paper cites.
Transfer learning for reinforcement learning domains: A survey
Matthew E. Taylor and Peter Stone · 2009
Earlier work this paper cites.
The interaction of maturational constraints and intrinsic motivations in active motor development
A. Baranes and P. Oudeyer · 2011
Earlier work this paper cites.
The strategic student approach for life-long exploration and learning
Manuel Lopes and Pierre-Yves Oudeyer · 2012
Earlier work this paper cites.
Active learning of inverse models with intrinsically motivated goal exploration in robots
Adrien Baranes and Pierre-Yves Oudeyer · 2013
Earlier work this paper cites.
How to discount deep reinforcement learning: Towards new dynamic strategies
Vincent François-Lavet et al · 2015
Earlier work this paper cites.
Language understanding for text-based games using deep reinforcement learning
Karthik Narasimhan et al · 2015
Earlier work this paper cites.
Unifying count-based exploration and intrinsic motivation
Marc Bellemare et al · 2016
Earlier work this paper cites.
Reinforcement learning with unsupervised auxiliary tasks
Max Jaderberg et al · 2016
Earlier work this paper cites.
Hindsight experience replay
Marcin Andrychowicz et al · 2017
Earlier work this paper cites.
Emergent complexity via multi-agent competition
Trapit Bansal et al · 2017
Earlier work this paper cites.
Reverse curriculum generation for reinforcement learning
Carlos Florensa et al · 2017
Earlier work this paper cites.
Intrinsically motivated goal exploration processes with automatic curriculum learning
Sébastien Forestier et al · 2017
Earlier work this paper cites.

Similar

Teacher-student curriculum learning
Tambet Matiisen et al · 2017
Cited alongside, same era.
Curiosity-driven exploration by self-supervised prediction
Deepak Pathak et al · 2017
Cited alongside, same era.
Robust adversarial reinforcement learning
Lerrel Pinto et al · 2017
Cited alongside, same era.
Mastering the game of go without human knowledge
David Silver et al · 2017
Cited alongside, same era.
Intrinsic motivation and automatic curricula via asymmetric self-play
Sainbayar Sukhbaatar et al · 2017
Cited alongside, same era.
Quantifying generalization in reinforcement learning
Karl Cobbe et al · 2018
Cited alongside, same era.
Mix & match agent curricula for reinforcement learning
Wojciech Czarnecki et al · 2018
Cited alongside, same era.
Curriculum goal masking for continuous deep reinforcement learning
Original
Manfred Eppe et al · 2018
Cited alongside, same era.
Diversity is all you need: Learning skills without a reward function
Benjamin Eysenbach et al · 2018
Cited alongside, same era.
Automatic goal generation for reinforcement learning agents
Carlos Florensa et al · 2018
Cited alongside, same era.
Accuracy-based curriculum learning in deep reinforcement learning
Pierre Fournier et al · 2018
Cited alongside, same era.
Distributed prioritized experience replay
Dan Horgan et al · 2018
Cited alongside, same era.

Then

CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning
Cédric Colas et al · 2019
Later among the works it cites.
Distilling policy distillation
Wojciech Marian Czarnecki et al · 2019
Later among the works it cites.
Curriculum-guided hindsight experience replay
Meng Fang et al · 2019
Later among the works it cites.
Reinforcement learning for improving agent design
David Ha · 2019
Later among the works it cites.
On the power of curriculum learning in training deep networks
Guy Hacohen and Daphna Weinshall · 2019
Later among the works it cites.
A generalized framework for self-play training
D. Hernandez et al · 2019
Later among the works it cites.
Unsupervised curricula for visual meta-reinforcement learning
Allan Jabri et al · 2019
Later among the works it cites.
Language grounding through social interactions and curiosity-driven multi-goal learning
Nicolas Lair et al · 2019
Later among the works it cites.
Continual learning for robotics
Timothée Lesort et al · 2019
Later among the works it cites.
Active domain randomization
Bhairav Mehta et al · 2019
Later among the works it cites.
Solving rubik’s cube with a robot hand
OpenAI et al · 2019
Later among the works it cites.
Self-supervised exploration via disagreement
Deepak Pathak et al · 2019
Later among the works it cites.
Skew-fit: State-covering self-supervised reinforcement learning
Vitchyr H. Pong et al · 2019
Later among the works it cites.
Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments
Rémy Portelas et al · 2019
Later among the works it cites.
Automated curricula through setter-solver interactions
Sébastien Racanière et al · 2019
Later among the works it cites.
Procedural content generation: From automatically generating game levels to increasing generality in machine learning
Sebastian Risi and Julian Togelius · 2019
Later among the works it cites.
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Oriol Vinyals et al · 2019
Later among the works it cites.
Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions
Rui Wang et al · 2019
Later among the works it cites.
Experience replay optimization
Daochen Zha et al · 2019
Later among the works it cites.
Curiosity-driven experience prioritization via density estimation
Rui Zhao and Volker Tresp · 2019
Later among the works it cites.
Language as a cognitive tool to imagine goals in curiosity-driven exploration
Cédric Colas et al · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…