2019

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Yarats, Denis, Zhang, Amy, Kostrikov, Ilya et al.

Understand

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult.

A promising approach is to learn a latent representation together with the control policy.
However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance.
Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning.

Built on

A markovian decision process
Bellman, R · 1957
Earlier work this paper cites.
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., and Cassandra, A. R · 1998
Earlier work this paper cites.
Deep auto-encoder neural networks in reinforcement learning
Lange, S. and Riedmiller, M. A · 2010
Earlier work this paper cites.
Learn to swing up and balance a real pole based on raw visual input data
Mattner, J., Lange, S., and Riedmiller, M · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Kingma, D. P. and Welling, M · 2013
Earlier work this paper cites.
Playing atari with deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M · 2013
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A. M., McClelland, J. L., and Ganguli, S · 2013
Earlier work this paper cites.
Learning visual feature spaces for robotic manipulation with deep spatial autoencoders
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., and Abbeel, P · 2015
Earlier work this paper cites.
Deep reinforcement learning with double q-learning
van Hasselt, H., Guez, A., and Silver, D · 2015
Earlier work this paper cites.
Layer normalization
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.

Similar

Asynchronous methods for deep reinforcement learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Learning state representation for deep actor-critic control
Munk, J., Kober, J., and Babuska, R · 2016
Cited alongside, same era.
Loss is its own reward: Self-supervision for reinforcement learning
Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T · 2016
Cited alongside, same era.
beta-vae: Learning basic visual concepts with a constrained variational framework
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A · 2017
Cited alongside, same era.
Reinforcement learning with unsupervised auxiliary tasks
Jaderberg, M., Mnih, V., Czarnecki, W., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K · 2017
Cited alongside, same era.
Visual interaction networks
Watters, N., Tacchetti, A., Weber, T., Pascanu, R., Battaglia, P. W., and Zoran, D · 2017
Cited alongside, same era.
Distributional policy gradients
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lillicrap, T · 2018
Cited alongside, same era.
Learning actionable representations from visual observations
Dwibedi, D., Tompson, J., Lynch, C., and Sermanet, P · 2018
Cited alongside, same era.
Recurrent world models facilitate policy evolution
Ha, D. and Schmidhuber, J · 2018
Cited alongside, same era.
Soft actor-critic algorithms and applications
Original
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al · 2018
Cited alongside, same era.
Darla: Improving zero-shot transfer in reinforcement learning
Higgins, I., Pal, A., Rusu, A. A., Matthey, L., Burgess, C. P., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A
Cited in the paper.
Decoupling dynamics and reward for transfer learning
Zhang, A., Satija, H., and Pineau, J
Cited in the paper.

Then

Learning latent dynamics for planning from pixels
Original
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J · 2018
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Later among the works it cites.
Visual reinforcement learning with imagined goals
Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S · 2018
Later among the works it cites.
Deepmind control suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al · 2018
Later among the works it cites.
Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks
Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S. S., and Pennington, J · 2018
Later among the works it cites.
Maximum entropy inverse reinforcement learning
Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K · 2018
Later among the works it cites.
Deepmdp: Learning continuous latent space models for representation learning
Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Bellemare, M. G · 2019
Closest in time.
From variational to deterministic autoencoders
Ghosh, P., Sajjadi, M. S. M., Vergari, A., Black, M. J., and Schölkopf, B · 2019
Closest in time.
Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model
Lee, A. X., Nagabandi, A., Abbeel, P., and Levine, S · 2019
Closest in time.
Soft actor-critic (sac) implementation in pytorch
Yarats, D. and Kostrikov, I · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…