2018

Kickstarting Deep Reinforcement Learning

Schmitt, Simon, Hudson, Jonathan J., Zidek, Augustin et al.

Understand

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent.

To this end, we leverage ideas from policy distillation and population based training.
Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance.
We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design.

Built on

Reinforcement learning: An introduction , volume 116
Sutton, R.S. and Barto, A.G · 1998
Earlier work this paper cites.
Model compression
Buciluǎ, Cristian, Caruana, Rich, and Niculescu-Mizil, Alexandru · 2006
Earlier work this paper cites.
A survey of robot learning from demonstration
Argall, Brenna D., Chernova, Sonia, Veloso, Manuela, and Browning, Brett · 2009
Earlier work this paper cites.
Toward an architecture for never-ending language learning
Carlson, Andrew, Betteridge, Justin, Kisiel, Bryan, Settles, Burr, Hruschka Jr, Estevam R, and Mitchell, Tom M · 2010
Earlier work this paper cites.
NEIL: Extracting visual knowledge from web data
Chen, Xinlei, Shrivastava, Abhinav, and Gupta, Abhinav · 2013
Earlier work this paper cites.
Do deep nets really need to be deep?
Ba, Jimmy and Caruana, Rich · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Girshick, Ross, Donahue, Jeff, Darrell, Trevor, and Malik, Jitendra · 2014
Earlier work this paper cites.
Learning and transferring mid-level image representations using convolutional neural networks
Oquab, Maxime, Bottou, Léon, Laptev, Ivan, and Sivic, Josef · 2014
Earlier work this paper cites.

Similar

The secret of our success: how culture is driving human evolution, domesticating our species, and making us smarter
Henrich, Joseph · 2015
Cited alongside, same era.
Distilling the knowledge in a neural network
Original
Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff · 2015
Cited alongside, same era.
Never ending learning
Mitchell, Tom M, Cohen, William W, Hruschka Jr, Estevam R, Talukdar, Partha Pratim, Betteridge, Justin, Carlson, Andrew, Mishra, Bhavana Dalvi, Gardner, Matthew, Kisiel, Bryan, Krishnamurthy, Jayant, et al · 2015
Cited alongside, same era.
Actor-mimic: Deep multitask and transfer reinforcement learning
Original
Parisotto, Emilio, Ba, Jimmy Lei, and Salakhutdinov, Ruslan · 2015
Cited alongside, same era.
Policy distillation
Original
Rusu, Andrei A., Colmenarejo, Sergio Gomez, Gülçehre, Çaglar, Desjardins, Guillaume, Kirkpatrick, James, Pascanu, Razvan, Mnih, Volodymyr, Kavukcuoglu, Koray, and Hadsell, Raia · 2015
Cited alongside, same era.
DeepMind Lab
Original
Beattie, Charles, Leibo, Joel Z., Teplyashin, Denis, Ward, Tom, Wainwright, Marcus, Kuttler, Heinrich, Lefrancq, Andrew, Green, Simon, Valdes, Victor, Sadik, Amir, Schrittwieser, Julian, Anderson, Keith, York, Sarah, Cant, Max, Cain, Adam, Bolton, Adrian, Gaffney, Stephen, King, Helen, Hassabis, Demis, Legg, Shane, and Petersen, Stig · 2016
Cited alongside, same era.
Mollifying networks
Original
Gulcehre, Caglar, Moczulski, Marcin, Visin, Francesco, and Bengio, Yoshua · 2016
Cited alongside, same era.

Then

Asynchronous methods for deep reinforcement learning
Mnih, Volodymyr, Badia, Adrià Puigdomènech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P., Harley, Tim, Silver, David, and Kavukcuoglu, Koray · 2016
Later among the works it cites.
Training recurrent neural networks by diffusion
Original
Mobahi, Hossein · 2016
Later among the works it cites.
Population based training of neural networks
Original
Jaderberg, Max, Dalibard, Valentin, Osindero, Simon, Czarnecki, Wojciech M., Donahue, Jeff, Razavi, Ali, Vinyals, Oriol, Green, Tim, Dunning, Iain, Simonyan, Karen, Fernando, Chrisantha, and Kavukcuoglu, Koray · 2017
Later among the works it cites.
Distral: Robust multitask reinforcement learning
Teh, Yee, Bapst, Victor, Czarnecki, Wojciech M, Quan, John, Kirkpatrick, James, Hadsell, Raia, Heess, Nicolas, and Pascanu, Razvan · 2017
Later among the works it cites.
Progressive reinforcement learning with distillation for multi-skilled motion control
Berseth, Glen, Xie, Cheng, Cernek, Paul, and Van de Panne, Michiel · 2018
Closest in time.
IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures
Original
Espeholt, Lasse, Soyer, Hubert, Munos, Remi, Simonyan, Karen, Mnih, Volodymir, Ward, Tom, Doron, Yotam, Firoiu, Vlad, Harley, Tim, Dunning, Iain, Legg, Shane, and Kavukcuoglu, Koray · 2018
Closest in time.
Psychlab: A psychology laboratory for deep reinforcement learning agents
Original
Leibo, Joel Z, d’Autume, Cyprien de Masson, Zoran, Daniel, Amos, David, Beattie, Charles, Anderson, Keith, Castañeda, Antonio García, Sanchez, Manuel, Green, Simon, Gruslys, Audrunas, et al · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…