2022

Adversarially Trained Actor Critic for Offline Reinforcement Learning

Cheng, Ching-An, Xie, Tengyang, Jiang, Nan et al.

Understand

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism.

ATAC is designed as a two-player Stackelberg game: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy.
We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters.
Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets.

Built on

Residual algorithms: Reinforcement learning with function approximation
Baird, L · 1995
Earlier work this paper cites.
Neuro-dynamic programming: an overview
Bertsekas, D. P. and Tsitsiklis, J. N · 1995
Earlier work this paper cites.
Stochastic approximation with two time scales
Borkar, V. S · 1997
Earlier work this paper cites.
Integral probability metrics and their generating classes of functions
Müller, A · 1997
Earlier work this paper cites.
Actor-critic algorithms
Konda, V. R. and Tsitsiklis, J. N · 2000
Earlier work this paper cites.
A natural policy gradient
Kakade, S. M · 2001
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Kakade, S. and Langford, J · 2002
Earlier work this paper cites.
Error bounds for approximate policy iteration
Munos, R · 2003
Earlier work this paper cites.
Td (0) converges provably faster than the residual gradient algorithm
Schoknecht, R. and Merke, A · 2003
Earlier work this paper cites.
Prediction, learning, and games
Cesa-Bianchi, N. and Lugosi, G · 2006
Earlier work this paper cites.
Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path
Antos, A., Szepesvári, C., and Munos, R · 2008
Earlier work this paper cites.
Finite-time bounds for fitted value iteration
Munos, R. and Szepesvári, C · 2008
Earlier work this paper cites.
Online markov decision processes
Even-Dar, E., Kakade, S. M., and Mansour, Y · 2009
Earlier work this paper cites.
Convergent temporal-difference learning with arbitrary smooth function approximation
Maei, H. R., Szepesvari, C., Bhatnagar, S., Precup, D., Silver, D., and Sutton, R. S · 2009
Earlier work this paper cites.
Error propagation for approximate policy and value iteration
Farahmand, A. M., Munos, R., and Szepesvári, C · 2010
Earlier work this paper cites.
Market structure and equilibrium
Von Stackelberg, H · 2010
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015
Earlier work this paper cites.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al · 2016
Earlier work this paper cites.

Similar

Wasserstein generative adversarial networks
Arjovsky, M., Chintala, S., and Bottou, L · 2017
Cited alongside, same era.
A unified view of entropy-regularized markov decision processes
Original
Neu, G., Jonsson, A., and Gómez, V · 2017
Cited alongside, same era.
Deeply aggrevated: Differentiable imitation learning for sequential prediction
Sun, W., Venkatraman, A., Gordon, G. J., Boots, B., and Bagnell, J. A · 2017
Cited alongside, same era.
Addressing function approximation error in actor-critic methods
Fujimoto, S., Hoof, H., and Meger, D · 2018
Cited alongside, same era.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S · 2018
Cited alongside, same era.
Reinforcement learning: An introduction
Sutton, R. S. and Barto, A. G · 2018
Cited alongside, same era.
Information-theoretic considerations in batch reinforcement learning
Chen, J. and Jiang, N · 2019
Cited alongside, same era.
Predictor-corrector policy optimization
Cheng, C.-A., Yan, X., Ratliff, N., and Boots, B · 2019
Cited alongside, same era.
Off-policy deep reinforcement learning without exploration
Fujimoto, S., Meger, D., and Precup, D · 2019
Cited alongside, same era.
A theory of regularized markov decision processes
Geist, M., Scherrer, B., and Pietquin, O · 2019
Cited alongside, same era.
Stabilizing off-policy q-learning via bootstrapping error reduction
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S · 2019
Cited alongside, same era.
Safe policy improvement with baseline bootstrapping
Laroche, R., Trichelair, P., and Des Combes, R. T · 2019
Cited alongside, same era.

Then

Provably good batch reinforcement learning without great exploration
Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E · 2020
Later among the works it cites.
Hyperparameter selection for offline reinforcement learning
Original
Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N · 2020
Later among the works it cites.
A game theoretic framework for model based reinforcement learning
Rajeswaran, A., Mordatch, I., and Kumar, V · 2020
Later among the works it cites.
Q* approximation schemes for batch reinforcement learning: A theoretical comparison
Xie, T. and Jiang, N · 2020
Later among the works it cites.
Mopo: Model-based offline policy optimization
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T · 2020
Later among the works it cites.
On the theory of policy gradient methods: Optimality, approximation, and distribution shift
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G · 2021
Later among the works it cites.
Heuristic-guided reinforcement learning
Cheng, C.-A., Kolobov, A., and Swaminathan, A · 2021
Later among the works it cites.
A minimalist approach to offline reinforcement learning
Fujimoto, S. and Gu, S. S · 2021
Later among the works it cites.
Is pessimism provably efficient for offline rl?
Jin, Y., Yang, Z., and Wang, Z · 2021
Later among the works it cites.
Offline reinforcement learning with implicit q-learning
Original
Kostrikov, I., Nair, A., and Levine, S · 2021
Later among the works it cites.
Of moments and matching: A game-theoretic framework for closing the imitation gap
Swamy, G., Choudhury, S., Bagnell, J. A., and Wu, S · 2021
Later among the works it cites.
Representation learning for online and offline rl in low-rank mdps
Original
Uehara, M., Zhang, X., and Sun, W · 2021
Later among the works it cites.
A convergent and efficient deep q network algorithm
Original
Wang, Z. T. and Ueda, M · 2021
Later among the works it cites.
Bellman-consistent pessimism for offline reinforcement learning
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A · 2021
Later among the works it cites.
Combo: Conservative offline model-based policy optimization
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C · 2021
Later among the works it cites.
Provable benefits of actor-critic methods for offline reinforcement learning
Zanette, A., Wainwright, M. J., and Brunskill, E · 2021
Later among the works it cites.
Towards hyperparameter-free policy selection for offline reinforcement learning
Zhang, S. and Jiang, N · 2021
Later among the works it cites.
Stackelberg actor-critic: Game-theoretic reinforcement learning algorithms
Original
Zheng, L., Fiez, T., Alumbaugh, Z., Chasnov, B., and Ratliff, L. J · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…