2022

Online Decision Transformer

Zheng, Qinqing, Zhang, Amy, Grover, Aditya

Understand

Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling.

However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment.
We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework.
Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning.

Built on

A markovian decision process
Bellman, R · 1957
Earlier work this paper cites.
Error bounds for approximate policy iteration
Munos, R · 2003
Earlier work this paper cites.
Convex optimization
Boyd, S., Boyd, S. P., and Vandenberghe, L · 2004
Earlier work this paper cites.
Error bounds for approximate value iteration
Munos, R · 2005
Earlier work this paper cites.
Conservative q-learning for offline reinforcement learning
Original
Kumar, A., Zhou, A., Tucker, G., and Levine, S · 2006
Earlier work this paper cites.
Accelerating online reinforcement learning with offline datasets
Original
Nair, A., Dalal, M., Gupta, A., and Levine, S · 2006
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Earlier work this paper cites.
Error propagation for approximate policy and value iteration
Farahmand, A. M., Munos, R., and Szepesvári, C · 2010
Earlier work this paper cites.
Deep auto-encoder neural networks in reinforcement learning
Lange, S. and Riedmiller, M · 2010
Earlier work this paper cites.
Counterfactual reasoning and learning systems: The example of computational advertising
Bottou, L., Peters, J., Quiñonero-Candela, J., Charles, D. X., Chickering, D. M., Portugaly, E., Ray, D., Simard, P., and Snelson, E · 2013
Earlier work this paper cites.
Guided policy search
Levine, S. and Koltun, V · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Scheduled sampling for sequence prediction with recurrent neural networks
Original
Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N · 2015
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Original
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al · 2015
Earlier work this paper cites.
Asynchronous methods for deep reinforcement learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Deep reinforcement learning with double q-learning
Van Hasselt, H., Guez, A., and Silver, D · 2016
Earlier work this paper cites.
Dueling network architectures for deep reinforcement learning
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N · 2016
Earlier work this paper cites.
Hindsight Experience Replay
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W · 2017
Earlier work this paper cites.

Similar

Hindsight policy gradients
Original
Rauber, P., Ummadisingu, A., Mutz, F., and Schmidhuber, J · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Automatic goal generation for reinforcement learning agents
Florensa, C., Held, D., Geng, X., and Abbeel, P · 2018
Cited alongside, same era.
Evaluating generalization in multiagent systems using agent-interaction graphs
Grover, A., Al-Shedivat, M., Gupta, J. K., Burda, Y., and Edwards, H · 2018
Cited alongside, same era.
Reinforcement learning and control as probabilistic inference: Tutorial and review
Original
Levine, S · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I · 2018
Cited alongside, same era.
Learning to reach goals via iterated supervised learning
Original
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S · 2019
Cited alongside, same era.
Model-based reinforcement learning for atari
Original
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al · 2019
Cited alongside, same era.
Stabilizing off-policy q-learning via bootstrapping error reduction
Original
Kumar, A., Fu, J., Tucker, G., and Levine, S · 2019
Cited alongside, same era.
Reinforcement learning upside down: Don’t predict rewards–just map them to actions
Original
Schmidhuber, J · 2019
Cited alongside, same era.
Training agents using upside-down reinforcement learning
Original
Srivastava, R. K., Shyam, P., Mutz, F., Jaśkowski, W., and Schmidhuber, J · 2019
Cited alongside, same era.

Then

Reset-free lifelong learning with skill-space planning
Original
Lu, K., Grover, A., Abbeel, P., and Mordatch, I · 2020
Later among the works it cites.
Stochastic optimization for performative prediction
Original
Mendler-Dünner, C., Perdomo, J. C., Zrnic, T., and Hardt, M · 2020
Later among the works it cites.
Performative prediction
Perdomo, J., Zrnic, T., Mendler-Dünner, C., and Hardt, M · 2020
Later among the works it cites.
Soft actor-critic (sac) implementation in pytorch
Yarats, D. and Kostrikov, I · 2020
Later among the works it cites.
Transfer learning with causal counterfactual reasoning in decision transformers
Original
Boustati, A., Chockler, H., and McNamee, D. C · 2021
Later among the works it cites.
Decision transformer: Reinforcement learning via sequence modeling
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I · 2021
Later among the works it cites.
Rvs: What is essential for offline rl via supervised learning?
Original
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S · 2021
Later among the works it cites.
A minimalist approach to offline reinforcement learning
Fujimoto, S. and Gu, S · 2021
Later among the works it cites.
Generalized decision transformer for offline hindsight information matching
Original
Furuta, H., Matsuo, Y., and Gu, S. S · 2021
Later among the works it cites.
Offline reinforcement learning as one big sequence modeling problem
Janner, M., Li, Q., and Levine, S · 2021
Later among the works it cites.
A survey of generalisation in deep reinforcement learning
Original
Kirk, R., Zhang, A., Grefenstette, E., and Rocktäschel, T · 2021
Later among the works it cites.
Should i run offline reinforcement learning or behavioral cloning?
Kumar, A., Hong, J., Singh, A., and Levine, S · 2021
Later among the works it cites.
Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J · 2021
Later among the works it cites.
AW-opt: Learning robotic skills with imitation andreinforcement at scale
Lu, Y., Hausman, K., Chebotar, Y., Yan, M., Jang, E., Herzog, A., Xiao, T., Irpan, A., Khansari, M., Kalashnikov, D., and Levine, S · 2021
Later among the works it cites.
Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks
Original
Meng, L., Wen, M., Yang, Y., Le, C., Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B · 2021
Later among the works it cites.
Shaking the foundations: delusions in sequence models for interaction and control
Original
Ortega, P. A., Kunesch, M., Delétang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot, B., Perolat, J., et al · 2021
Later among the works it cites.
It takes four to tango: Multiagent selfplay for automatic curriculum generation
Original
Du, Y., Abbeel, P., and Grover, A · 2022
Closest in time.
Pretrained transformers as universal computation engines
Lu, K., Grover, A., Abbeel, P., and Mordatch, I · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…