2020

Variational Policy Gradient Method for Reinforcement Learning with General Utilities

Zhang, Junyu, Koppel, Alec, Bedi, Amrit Singh et al.

Understand

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences.

In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general concave utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases.
Such generality invalidates the Bellman equation.
As this means that dynamic programming no longer works, we focus on direct policy search.

Built on

A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
Stochastic estimation of the maximum of a regression function
Jack Kiefer, Jacob Wolfowitz, et al · 1952
Earlier work this paper cites.
Studies in Linear and Non-Linear Programming , volume II of Stanford Mathematical Studies in the Social Sciences
K.J. Arrow, L. Hurwicz, and H. Uzawa · 1958
Earlier work this paper cites.
Some remarks on finite horizon Markovian decision models
Cyrus Derman and Morton Klein · 1965
Earlier work this paper cites.
Non-Markovian processes
L Takács · 1966
Earlier work this paper cites.
Linear Programming and Finite Markovian Control Problems
L C M Kallenberg · 1983
Earlier work this paper cites.
Learning to predict by the methods of temporal differences
Richard S Sutton · 1988
Earlier work this paper cites.
Variance-penalized Markov decision processes
Jerzy A. Filar, L. C. M. Kallenberg, and Huey-Miin Lee · 1989
Earlier work this paper cites.
Q-learning
Christopher JCH Watkins and Peter Dayan · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
On finding optimal policies for Markov decision chains: A unifying framework for mean-variance-tradeoffs
Ying Huang and L. C. M. Kallenberg · 1994
Earlier work this paper cites.
Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory
L. C. M. Kallenberg · 1994
Earlier work this paper cites.
Reinforcement learning of non-Markov decision processes
Steven D Whitehead and Long-Ji Lin · 1995
Earlier work this paper cites.
Learning from demonstration
Stefan Schaal · 1997
Earlier work this paper cites.
Constrained Markov decision processes , volume 7
Eitan Altman · 1999
Earlier work this paper cites.
Actor-critic–type learning algorithms for Markov Decision Processes
Vijaymohan R Konda and Vivek S Borkar · 1999
Earlier work this paper cites.
Actor-critic algorithms
Vijay R Konda and John N Tsitsiklis · 2000
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour · 2000
Earlier work this paper cites.

Similar

Risk-sensitive optimal control for Markov decision processes with monotone cost
Vivek S Borkar and Sean P Meyn · 2002
Cited alongside, same era.
A natural policy gradient
Sham M Kakade · 2002
Cited alongside, same era.
Approximately optimal approximate reinforcement learning
J Langford S Kakade · 2002
Cited alongside, same era.
Cautious reinforcement learning via distributional risk in the dual domain
Original
Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel · 2002
Cited alongside, same era.
Numerical optimization
Jorge Nocedal and Stephen Wright · 2006
Cited alongside, same era.
Generalization bounds for stochastic saddle point problems
Original
Junyu Zhang, Mingyi Hong, Mengdi Wang, and shuzhong Zhang · 2006
Cited alongside, same era.
Stochastic approximation: A dynamical systems viewpoint
Vivek S Borkar · 2008
Cited alongside, same era.
Introduction to nonparametric estimation
Alexandre B Tsybakov · 2008
Cited alongside, same era.
A survey of robot learning from demonstration
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning · 2009
Cited alongside, same era.
Natural actor-critic algorithms
Shalabh Bhatnagar, Richard Sutton, Mohammad Ghavamzadeh, and Mark Lee · 2009
Cited alongside, same era.
Stochastic Approximation: A Dynamical Systems Viewpoint , volume 77
Vivek S. Borkar · 2009
Cited alongside, same era.
A general projection property for distribution families
Y.-L. Yu, Y. Li, D. Schuurmans, and Cs. Szepesvári · 2009
Cited alongside, same era.

Then

Policy gradient in Lipschitz Markov Decision Processes
Matteo Pirotta, Marcello Restelli, and Luca Bascetta · 2015
Later among the works it cites.
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Later among the works it cites.
Openai gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Later among the works it cites.
Constrained policy optimization
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Later among the works it cites.
Global convergence of policy gradient methods for the linear quadratic regulator
Maryam Fazel, Rong Ge, Sham M Kakade, and Mehran Mesbahi · 2018
Later among the works it cites.
Provably efficient maximum entropy exploration
Original
Elad Hazan, Sham M Kakade, Karan Singh, and Abby Van Soest · 2018
Later among the works it cites.
Stochastic variance-reduced policy gradient
Original
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli · 2018
Later among the works it cites.
Optimality and approximation with policy gradient methods in Markov decision processes
Original
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan · 2019
Later among the works it cites.
Global optimality guarantees for policy gradient methods
Original
Jalaj Bhandari and Daniel Russo · 2019
Later among the works it cites.
LQR through the lens of first order methods: Discrete-time case
Original
Jingjing Bu, Afshin Mesbahi, Maryam Fazel, and Mehran Mesbahi · 2019
Later among the works it cites.
Efficiency of minimizing compositions of convex functions and smooth maps
Dmitriy Drusvyatskiy and Courtney Paquette · 2019
Later among the works it cites.
Neural proximal/trust region policy optimization attains globally optimal policy
Original
Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang · 2019
Later among the works it cites.
Neural policy gradient methods: Global optimality and rates of convergence
Original
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang · 2019
Later among the works it cites.
Sample efficient policy gradient methods with recursive variance reduction
Original
Pan Xu, Felicia Gao, and Quanquan Gu · 2019
Later among the works it cites.
Global convergence of policy gradient methods to (almost) locally optimal policies
Original
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Başar · 2019
Later among the works it cites.
Global optimality guarantees for policy gradient methods
Jalaj Bhandari and Daniel Russo · 2020
Closest in time.
On the global convergence rates of softmax policy gradient methods
Original
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…