2017

Policy Optimization by Genetic Distillation

Gangwani, Tanmay, Peng, Jian

Understand

Genetic algorithms have been widely used in many practical optimization problems.

Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization.
However, they have not been shown useful for deep reinforcement learning, possibly due to the catastrophic consequence of parameter crossovers of neural networks.
Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sample-efficient deep policy optimization.

Built on

Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Efficient reinforcement learning through symbiotic evolution
David E Moriarty and Risto Mikkulainen · 1996
Earlier work this paper cites.
Solving non-markovian control tasks with neuroevolution
Faustino J Gomez and Risto Miikkulainen · 1999
Earlier work this paper cites.
Completely derandomized self-adaptation in evolution strategies
Nikolaus Hansen and Andreas Ostermeier · 2001
Earlier work this paper cites.
Neuroevolution for reinforcement learning using evolution strategies
Christian Igel · 2003
Earlier work this paper cites.
Learning tetris using the noisy cross-entropy method
István Szita and András Lörincz · 2006
Earlier work this paper cites.
A novel generative encoding for exploiting neural network sensor and output geometry
David B D’Ambrosio and Kenneth O Stanley · 2007
Earlier work this paper cites.
Accelerated neural evolution through cooperatively coevolved synapses
Faustino Gomez, Jürgen Schmidhuber, and Risto Miikkulainen · 2008
Earlier work this paper cites.
Visualizing data using t-sne
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.

Similar

Neuroevolution strategies for episodic reinforcement learning
Verena Heidrich-Meisner and Christian Igel · 2009
Cited alongside, same era.
Parameter-exploring policy gradients
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber · 2010
Cited alongside, same era.
A reduction of imitation learning and structured prediction to no-regret online learning
Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell · 2011
Cited alongside, same era.
Evolutionary computation for reinforcement learning
Shimon Whiteson · 2012
Cited alongside, same era.
A neuroevolution approach to general atari game playing
Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone · 2014
Cited alongside, same era.
Continuous control with deep reinforcement learning
Original
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Cited alongside, same era.
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Cited alongside, same era.
Benchmarking deep reinforcement learning for continuous control
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel · 2016
Cited alongside, same era.
Efficient reinforcement learning through evolving neural network topologies
Kenneth O Stanley and Risto Miikkulainen
Cited in the paper.
Evolving neural networks through augmenting topologies
Kenneth O Stanley and Risto Miikkulainen
Cited in the paper.
Reinforcement learning: An introduction , volume 1
Richard S Sutton and Andrew G Barto
Cited in the paper.

Then

End-to-end training of deep visuomotor policies
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel · 2016
Later among the works it cites.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Later among the works it cites.
Tensorflow agents: Efficient batched reinforcement learning in tensorflow
Original
Danijar Hafner, James Davidson, and Vincent Vanhoucke · 2017
Closest in time.
Emergence of locomotion behaviours in rich environments
Original
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al · 2017
Closest in time.
Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients
J. Lehman, J. Chen, J. Clune, and K. O. Stanley · 2017
Closest in time.
Evolution strategies as a scalable alternative to reinforcement learning
Original
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever · 2017
Closest in time.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Closest in time.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…