2016

Combining policy gradient and Q-learning

O'Donoghue, Brendan, Munos, Remi, Kavukcuoglu, Koray et al.

Understand

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting.

However, vanilla online variants are on-policy only and not able to take advantage of off-policy data.
In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer.
This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values.

Built on

Dynamic programming
Richard Bellman · 1957
Earlier work this paper cites.
Learning to predict by the methods of temporal differences
Richard S Sutton · 1988
Earlier work this paper cites.
Learning from delayed rewards
Christopher John Cornish Hellaby Watkins · 1989
Earlier work this paper cites.
Function optimization using connectionist reinforcement learning algorithms
Ronald J Williams and Jing Peng · 1991
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Advantage updating
Leemon C Baird III · 1993
Earlier work this paper cites.
Reinforcement learning for robots using neural networks
Long-Ji Lin · 1993
Earlier work this paper cites.
On-line Q-learning using connectionist systems
Gavin A Rummery and Mahesan Niranjan · 1994
Earlier work this paper cites.
Temporal difference learning and TD-Gammon
Gerald Tesauro · 1995
Earlier work this paper cites.
Neuro-Dynamic Programming
Dimitri P. Bertsekas and John N. Tsitsiklis · 1996
Earlier work this paper cites.
Incremental multi-step Q-learning
Jing Peng and Ronald J Williams · 1996
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-Ichi Amari · 1998
Earlier work this paper cites.
Reinforcement Learning: an Introduction
R. Sutton and A. Barto · 1998
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al · 1999
Earlier work this paper cites.
A natural policy gradient
Sham Kakade · 2001
Earlier work this paper cites.
Sequential cost-sensitive decision making with reinforcement learning
Edwin Pednault, Naoki Abe, and Bianca Zadrozny · 2002
Earlier work this paper cites.

Similar

Covariant policy search
J Andrew Bagnell and Jeff Schneider · 2003
Cited alongside, same era.
On actor-critic algorithms
Vijay R Konda and John N Tsitsiklis · 2003
Cited alongside, same era.
Reinforcement learning with factored states and actions
Brian Sallans and Geoffrey E Hinton · 2004
Cited alongside, same era.
Dynamic programming and optimal control , volume 1
Dimitri P Bertsekas · 2005
Cited alongside, same era.
Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method
Martin Riedmiller · 2005
Cited alongside, same era.
A theoretical and empirical analysis of expected sarsa
Harm Van Seijen, Hado Van Hasselt, Shimon Whiteson, and Marco Wiering · 2009
Cited alongside, same era.
Relative entropy policy search
Jan Peters, Katharina Mülling, and Yasemin Altun · 2010
Cited alongside, same era.
Dynamic policy programming
Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J Kappen · 2012
Cited alongside, same era.
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2012
Cited alongside, same era.
Off-policy actor-critic
Thomas Degris, Martha White, and Richard S Sutton · 2012
Cited alongside, same era.
Actor-critic reinforcement learning with energy-based policies
Nicolas Heess, David Silver, and Yee Whye Teh · 2012
Cited alongside, same era.
Guided policy search
Sergey Levine and Vladlen Koltun · 2013
Cited alongside, same era.

Then

Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Later among the works it cites.
Bias in natural actor-critic algorithms
Philip Thomas · 2014
Later among the works it cites.
Taming the noise in reinforcement learning via soft updates
Original
Roy Fox, Ari Pakman, and Naftali Tishby · 2015
Later among the works it cites.
Policy gradient methods for off-policy control
Original
Lucas Lehnert and Doina Precup · 2015
Later among the works it cites.
End-to-end training of deep visuomotor policies
Original
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel · 2015
Later among the works it cites.
Continuous control with deep reinforcement learning
Original
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2015
Later among the works it cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Later among the works it cites.
Prioritized experience replay
Original
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver · 2015
Later among the works it cites.
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Later among the works it cites.
On-policy vs. off-policy updates for deep reinforcement learning
Matthew Hausknecht and Peter Stone · 2016
Closest in time.
Asynchronous methods for deep reinforcement learning
Original
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Closest in time.
Reward augmented maximum likelihood for neural structured prediction
Original
Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans · 2016
Closest in time.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Closest in time.
Deep reinforcement learning with double Q-learning
Hado Van Hasselt, Arthur Guez, and David Silver · 2016
Closest in time.
Dueling network architectures for deep reinforcement learning
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…