2018

Soft-Robust Actor-Critic Policy-Gradient

Derman, Esther, Mankowitz, Daniel J., Mann, Timothy A. et al.

Understand

Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems.

However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative.
Our soft-robust framework is an attempt to overcome this issue.
In this paper, we present a novel Soft-Robust Actor-Critic algorithm (SR-AC).

Built on

Stochastic Approximation Methods for Constrained and Unconstrained Systems
Harold J. Kushner and Dean S. Clark · 1978
Earlier work this paper cites.
Innovation in conservative and entrepreneurial firms: Two models of strategic momentum
Danny Miller and Peter H. Friesen · 1982
Earlier work this paper cites.
Neuronlike adaptive elements that can solve difficult learning control problems
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson · 1983
Earlier work this paper cites.
Q-learning
Christopher J. C. H. Watkins and Peter Dayan · 1992
Earlier work this paper cites.
Stochastic Approximation with Two Timescales
Vivek S. Borkar · 1997
Earlier work this paper cites.
Stochastic Approximation Algorithms and Applications
Harold J. Kushner and G. George Yin · 1997
Earlier work this paper cites.
Actor-Critic Algorithms
Vijay R. Konda and John N. Tsitsiklis · 2000
Earlier work this paper cites.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
Richard S. Sutton, David McAllester, Satinger Singh, and Yishay Mansour · 2000
Earlier work this paper cites.
Robust Dynamic Programming
Garud N. Iyengar · 2005
Earlier work this paper cites.
Robust control of Markov decision processes with uncertain transition matrices
Arnab Nilim and Laurent El Ghaoui · 2005
Earlier work this paper cites.
Bayesian Actor-Critic Algorithms
Mohammad Ghavamzadeh and Yaakov Engel · 2007
Earlier work this paper cites.
Bias and Variance Approximation in Value Function Estimates
Shie Mannor, Duncan Simester, Peng Sun, and John N. Tsitsiklis · 2007
Earlier work this paper cites.

Similar

Natural Actor-Critic Algorithms
Shalabh Bhatnagar, Richard Sutton, Mohammad Ghavamzadeh, and Mark Lee · 2009
Cited alongside, same era.
Markov decision processes: Discrete stochastic dynamic programming , volume 414
Martin L. Puterman · 2009
Cited alongside, same era.
A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients
Ivo Grondman, Lucian Busoniu, Gabriel A.D. Lopes, and Robert Babuska · 2012
Cited alongside, same era.
Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty
Shie Mannor, Ofir Mebel, and Huan Xu · 2012
Cited alongside, same era.
Reinforcement Learning: State-of-the-Art
Marco Wiering and Martijn van Otterlo · 2012
Cited alongside, same era.
Distributionally Robust Markov Decision Processes
Huan Xu and Shie Mannor · 2012
Cited alongside, same era.
The Market for Retirement Financial Advice
Olivia S. Mitchell and Kent Smetters · 2013
Cited alongside, same era.
Playing Atari with Deep Reinforcement Learning: Technical Report
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller · 2013
Cited alongside, same era.
Deterministic Policy Gradient Algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Cited alongside, same era.
Scaling up robust mdps using function approximation
Aviv Tamar, Shie Mannor, and Huan Xu · 2014
Cited alongside, same era.
Bayesian Reinforcement Learning: A Survey
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar · 2015
Cited alongside, same era.

Then

Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Later among the works it cites.
Policy gradient for coherent risk measures
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor · 2015
Later among the works it cites.
OpenAI Gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Later among the works it cites.
Bayesian Policy Gradient and Actor-Critic Algorithms
Mohammad Ghavamzadeh, Yaakov Engel, and Michal Valko · 2016
Later among the works it cites.
Continuous Control with Deep Reinforcement Learning
Original
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2016
Later among the works it cites.
Reinforcement Learning in Robust Markov Decision Processes
Shiau Hong Lim, Huan Xu, and Shie Mannor · 2016
Later among the works it cites.
Robust MDPs with k-Rectangular Uncertainty
Shie Mannor, Ofir Mebel, and Huan Xu · 2016
Later among the works it cites.
Distributionally Robust Counterpart in Markov Decision Processes
Pengqian Yu and Huan Xu · 2016
Later among the works it cites.
Deep Robust Kalman Filter
Original
Shirli Di-Castro Shashua and Shie Mannor · 2017
Later among the works it cites.
Reinforcement learning under Model Mismatch
Aurko Roy, Huan Xu, and Sebastian Pokutta · 2017
Later among the works it cites.
Learning Robust Options
Daniel J Mankowitz, Timothy A Mann, Shie Mannor, Doina Precup, and Pierre-Luc Bacon · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…