2019

Lyapunov-based Safe Policy Optimization for Continuous Control

Chow, Yinlam, Nachum, Ofir, Faust, Aleksandra et al.

Understand

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations.

We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them.
Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints.
Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data.

Built on

Noninear systems
Khalil, H · 1996
Earlier work this paper cites.
Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program
Altman, E · 1998
Earlier work this paper cites.
Constrained Markov decision processes , volume 7
Altman, E · 1999
Earlier work this paper cites.
Nonlinear programming
Bertsekas, D · 1999
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Sutton, R., McAllester, D., Singh, S., and Mansour, Y · 2000
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Kakade, S. and Langford, J · 2002
Earlier work this paper cites.
Lyapunov design for safe reinforcement learning
Perkins, T. and Barto, A · 2002
Earlier work this paper cites.
Dynamic programming and optimal control , volume 1-2
Bertsekas, D · 2005
Earlier work this paper cites.
Risk-sensitive reinforcement learning applied to control under constraints
Geibel, P. and Wysotzki, F · 2005
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y · 2012
Earlier work this paper cites.
Playing atari with deep reinforcement learning
Original
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M · 2013
Earlier work this paper cites.

Similar

Continuous action reinforcement learning for control-affine systems with unknown dynamics
Faust, A., Ruymgaart, P., Salman, M., Fierro, R., and Tapia, L · 2014
Cited alongside, same era.
Continuous control with deep reinforcement learning
Original
Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Cited alongside, same era.
Prioritized experience replay
Original
Schaul, T., Quan, J., Antonoglou, I., and Silver, D · 2015
Cited alongside, same era.
End-to-end training of deep visuo-motor policies
Levine, S., Finn, C., Darrell, T., and Abbeel, P · 2016
Cited alongside, same era.
Continuous control with deep reinforcement learning
Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2016
Cited alongside, same era.
Fetch & freight: Standard platforms for service robot applications
Melonee Wise, Michael Ferguson, D. K. E. D. and Dymesich, D · 2016
Cited alongside, same era.
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D · 2016
Cited alongside, same era.
Constrained policy optimization
Original
Achiam, J., Held, D., Tamar, A., and Abbeel, P · 2017
Cited alongside, same era.
Optnet: Differentiable optimization as a layer in neural networks
Original
Amos, B. and Kolter, Z · 2017
Cited alongside, same era.
Trust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M., and Abbeel, P
Cited in the paper.
High-dimensional continuous control using generalized advantage estimation
Original
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P
Cited in the paper.

Then

Safe model-based reinforcement learning with stability guarantees
Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A · 2017
Later among the works it cites.
Risk-constrained reinforcement learning with percentile risk criteria
Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M · 2017
Later among the works it cites.
TensorFlow Agents: Efficient batched reinforcement learning in tensorflow
Original
Hafner, D., Davidson, J., and Vanhoucke, V · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Later among the works it cites.
A Lyapunov-based approach to safe reinforcement learning
Chow, Y., Nachum, O., Ghavamzadeh, M., and Duenez-Guzman, E · 2018
Later among the works it cites.
Safe exploration in continuous action spaces
Original
Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y · 2018
Later among the works it cites.
PRM-RL: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning
Original
Faust, A., Ramirez, O., Fiser, M., Oslund, K., Francis, A., Davidson, J., and Tapia, L · 2018
Later among the works it cites.
QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation
Original
Kalashnikov, D., Irpan, A., Sampedro, P. P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., and Levine, S · 2018
Later among the works it cites.
A general projection property for distribution families
Yu, Y., Li, Y., Schuurmans, D., and Szepesvári, C · 2018
Later among the works it cites.
Learning navigation behaviors end to end with autorl
Original
Chiang, H. L., Faust, A., Fiser, M., and Francis, A · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…