2015

Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies

Balduzzi, David, Ghifary, Muhammad

Understand

This paper proposes GProp, a deep reinforcement learning algorithm for continuous policies with compatible function approximation.

The algorithm is based on two innovations.
Firstly, we present a temporal-difference based method for learning the gradient of the value-function.
Secondly, we present the deviator-actor-critic (DAC) model, which comprises three neural networks that estimate the value function, its gradient, and determine the actor's policy respectively.

Built on

Neuronlike Adapative Elements That Can Solve Difficult Learning Control Problems
Andrew G Barto, Richard S Sutton, and Charles W Anderson · 1983
Earlier work this paper cites.
Problem complexity and method efficiency in optimization
A S Nemirovski and D B Yudin · 1983
Earlier work this paper cites.
Learning to control an unstable system with forward modeling
Michael I Jordan and R A Jacobs · 1990
Earlier work this paper cites.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Ronald J Williams · 1992
Earlier work this paper cites.
Residual algorithms: Reinforcement learning with function approximation
L C Baird · 1995
Earlier work this paper cites.
Adaptive Critic Designs
Daniel V Prokhorov and Donald C Wunsch · 1997
Earlier work this paper cites.
An Analysis of Temporal-Difference Learning with Function Approximation
John Tsitsiklis and Benjamin Van Roy · 1997
Earlier work this paper cites.
Reinforcement Learning: An Introduction
R S Sutton and A G Barto · 1998
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour · 1999
Earlier work this paper cites.
Actor-critic algorithms
Vijay R Konda and John N Tsitsiklis · 2000
Earlier work this paper cites.
A natural policy gradient
Sham Kakade · 2001
Earlier work this paper cites.
On-line learning control by association and reinforcement
Y Wang and J Si · 2001
Earlier work this paper cites.
Coordinated Reinforcement Learning
Carlos Guestrin, Michail Lagoudakis, and Ronald Parr · 2002
Earlier work this paper cites.
Unifying Temporal and Structural Credit Assignment Problems
Adrian K Agogino and Kagan Tumer · 2004
Earlier work this paper cites.
Learning to control an octopus arm with gaussian process temporal difference methods
Y Engel, P Szabó, and D Volkinshtein · 2005
Earlier work this paper cites.

Similar

Online convex optimization in the bandit setting: Gradient descent without a gradient
Abraham Flaxman, Adam Kalai, and H Brendan McMahan · 2005
Cited alongside, same era.
Policy Gradient Methods for Robotics
Jan Peters and Stefan Schaal · 2006
Cited alongside, same era.
Analyzing and Visualizing Multiagent Rewards in Dynamic and Stochastic Environments
Adrian K Agogino and Kagan Tumer · 2008
Cited alongside, same era.
Local Gaussian Process Regression for Real Time Online Model Learning
Duy Nguyen-Tuong, Jan Peters, and Matthias Seeger · 2008
Cited alongside, same era.
Theano: A CPU and GPU Math Expression Compiler
J Bergstra, O Breuleux, F Bastien, P Lamblin, R Pascanu, G Desjardins, J Turian, D Warde-Farley, and Yoshua Bengio · 2010
Cited alongside, same era.
Rectified Linear Units Improve Restricted Boltzmann Machines
Vinod Nair and Geoffrey Hinton · 2010
Cited alongside, same era.
A Survey on Policy Search for Robotics
Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters · 2011
Cited alongside, same era.
Deep Sparse Rectifier Neural Networks
Xavier Glorot, Antoine Bordes, and Yoshua Bengio · 2011
Cited alongside, same era.
Reinforcement learning in feedback control: Challenges and benchmarks from technical process control
Roland Hafner and Martin Riedmiller · 2011
Cited alongside, same era.
Information-Based Complexity, Feedback and Dynamics in Convex Programming
Maxim Raginsky and Alexander Rakhlin · 2011
Cited alongside, same era.
Theano: new features and speed improvements
F Bastien, P Lamblin, R Pascanu, J Bergstra, I Goodfellow, A Bergeron, N Bouchard, and Y Bengio · 2012
Cited alongside, same era.
Value-Gradient Learning
Michael Fairbank and Eduardo Alonso · 2012
Cited alongside, same era.

Then

An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time
Michael Fairbank, Eduardo Alonso, and Daniel V Prokhorov · 2013
Later among the works it cites.
Gradient Weights help Nonparametric Regressors
Samory Kpotufe and Abdeslam Boularias · 2013
Later among the works it cites.
On Rectified Linear Units for Speech Processing
M D Zeiler, M Ranzato, R Monga, M Mao, K Yang, Q V Le, P Nguyen, A Senior, V Vanhoucke, J Dean, and G Hinton · 2013
Later among the works it cites.
Policy Evaluation with Temporal Differences: A Survey and Comparison
Christoph Dann, Gerhard Neumann, and Jan Peters · 2014
Later among the works it cites.
Doubly Robust Policy Evaluation and Optimization
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li · 2014
Later among the works it cites.
Combining Reward Shaping and Hierarchies for Scaling to Large Multiagent Systems
Chris HolmesParker, Adrian K Agogino, and Kagan Tumer · 2014
Later among the works it cites.
Deterministic Policy Gradient Algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Later among the works it cites.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Later among the works it cites.
A Consistent Estimator of the Expected Gradient Outerproduct
Shubhendu Trivedi, Jialei Wang, Samory Kpotufe, and Gregory Shakhnarovich · 2014
Later among the works it cites.
Deep Online Convex Optimization by Putting Forecaster to Sleep
Original
David Balduzzi · 2015
Closest in time.
Kickback cuts Backprop’s red-tape: Biologically plausible credit assignment in neural networks
David Balduzzi, Hastagiri Vanchinathan, and Joachim Buhmann · 2015
Closest in time.
End-to-End Training of Deep Visuomotor Policies
Original
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel · 2015
Closest in time.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Closest in time.
From Pixels to Torques: Policy Learning with Deep Dynamical Models
Original
Niklas Wahlström, Thomas B. Schön, and Marc Peter Deisenroth · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…