2021

Decoupling Value and Policy for Generalization in Reinforcement Learning

Raileanu, Roberta, Fergus, Rob

Understand

Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images.

However, we argue that more information is needed to accurately estimate the value function than to learn the optimal policy.
Consequently, the use of a shared representation for the policy and value function can lead to overfitting.
To alleviate this problem, we propose two approaches which are combined to create IDAAC: Invariant Decoupled Advantage Actor-Critic.

Built on

Policy gradient methods for reinforcement learning with function approximation
Sutton, R., McAllester, D. A., Singh, S., and Mansour, Y · 1999
Earlier work this paper cites.
Invariant causal prediction for block mdps
Original
Zhang, A., Lyle, C., Sodhani, S., Filos, A., Kwiatkowska, M., Pineau, J., Gal, Y., and Precup, D · 2003
Earlier work this paper cites.
Learning invariant representations for reinforcement learning without reconstruction
Original
Zhang, A., McAllister, R., Calandra, R., Gal, Y., and Levine, S · 2006
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Trust region policy optimization
Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P · 2015
Earlier work this paper cites.
Asynchronous methods for deep reinforcement learning
Original
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K · 2016
Earlier work this paper cites.
Dueling network architectures for deep reinforcement learning
Original
Wang, Z., Schaul, T., Hessel, M., Hasselt, H. V., Lanctot, M., and Freitas, N. D · 2016
Earlier work this paper cites.
Unsupervised learning of disentangled representations from video
Denton, E. L. and Birodkar, V · 2017
Earlier work this paper cites.
The kinetics human action video dataset
Original
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, A., Suleyman, M., and Zisserman, A · 2017
Earlier work this paper cites.
Towards generalization and simplicity in continuous control
Original
Rajeswaran, A., Lowrey, K., Todorov, E., and Kakade, S. M · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
Mastering the game of go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D · 2017
Earlier work this paper cites.
A study on overfitting in deep reinforcement learning
Original
Zhang, C., Vinyals, O., Munos, R., and Bengio, S · 2017
Earlier work this paper cites.
Distributed distributional deterministic policy gradients
Original
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Dhruva, T., Muldal, A., Heess, N., and Lillicrap, T · 2018
Earlier work this paper cites.
Quantifying generalization in reinforcement learning
Original
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J · 2018
Earlier work this paper cites.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures
Original
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al · 2018
Earlier work this paper cites.

Similar

Generalization and regularization in dqn
Original
Farebrother, J., Machado, M. C., and Bowling, M. H · 2018
Cited alongside, same era.
Illuminating generalization in deep reinforcement learning through procedural level generation
Justesen, N., Torrado, R. R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S · 2018
Cited alongside, same era.
Pytorch implementations of reinforcement learning algorithms
Kostrikov, I · 2018
Cited alongside, same era.
Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M. J., and Bowling, M. H · 2018
Cited alongside, same era.
Gotta learn fast: A new benchmark for generalization in rl
Original
Nichol, A., Pfau, V., Hesse, C., Klimov, O., and Schulman, J · 2018
Cited alongside, same era.
Assessing generalization in deep reinforcement learning
Original
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., and Song, D. X · 2018
Cited alongside, same era.
Asymmetric actor critic for image-based robot learning
Original
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., and Abbeel, P · 2018
Cited alongside, same era.
Deepmind control suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. A · 2018
Cited alongside, same era.
Invariant risk minimization
Original
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D · 2019
Cited alongside, same era.
Leveraging procedural generation to benchmark reinforcement learning
Original
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J · 2019
Cited alongside, same era.
Transfer learning for related reinforcement learning tasks via image-to-image translation
Original
Gamrian, S. and Goldberg, Y · 2019
Cited alongside, same era.
Generalization in reinforcement learning with selective noise injection and information bottleneck
Igl, M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C., Devlin, S., and Hofmann, K · 2019
Cited alongside, same era.

Then

Phasic policy gradient
Original
Cobbe, K., Hilton, J., Klimov, O., and Schulman, J · 2020
Later among the works it cites.
Measuring visual generalization in continuous control from pixels
Original
Grigsby, J. and Qi, Y · 2020
Later among the works it cites.
The impact of non-stationarity on generalisation in deep reinforcement learning
Original
Igl, M., Farquhar, G., Luketina, J., Böhmer, W., and Whiteson, S · 2020
Later among the works it cites.
Prioritized level replay
Original
Jiang, M., Grefenstette, E., and Rocktäschel, T · 2020
Later among the works it cites.
Image augmentation is all you need: Regularizing deep reinforcement learning from pixels
Original
Kostrikov, I., Yarats, D., and Fergus, R · 2020
Later among the works it cites.
The nethack learning environment
Original
Kuttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T · 2020
Later among the works it cites.
Reinforcement learning with augmented data
Original
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A · 2020
Later among the works it cites.
Network randomization: A simple technique for generalization in deep reinforcement learning
Lee, K., Lee, K., Shin, J., and Lee, H · 2020
Later among the works it cites.
Deep reinforcement and infomax learning
Original
Mazoure, B., des Combes, R. T., Doan, T., Bachman, P., and Hjelm, R. D · 2020
Later among the works it cites.
Ride: Rewarding impact-driven exploration for procedurally-generated environments
Original
Raileanu, R. and Rocktäschel, T · 2020
Later among the works it cites.
Automatic data augmentation for generalization in deep reinforcement learning
Original
Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., and Fergus, R · 2020
Later among the works it cites.
Visual transfer for reinforcement learning via wasserstein domain confusion
Original
Roy, J. and Konidaris, G · 2020
Later among the works it cites.
Invariant policy optimization: Towards stronger generalization in reinforcement learning
Original
Sonar, A., Pacelli, V., and Majumdar, A · 2020
Later among the works it cites.
Observational overfitting in reinforcement learning
Original
Song, X., Jiang, Y., Tu, S., Du, Y., and Neyshabur, B · 2020
Later among the works it cites.
Decoupling representation learning from reinforcement learning
Original
Stooke, A., Lee, K., Abbeel, P., and Laskin, M · 2020
Later among the works it cites.
Improving generalization in reinforcement learning with mixture regularization
Original
Wang, K., Kang, B., Shao, J., and Feng, J · 2020
Later among the works it cites.
Rotation, translation, and cropping for zero-shot generalization
Original
Ye, C., Khalifa, A., Bontrager, P., and Togelius, J · 2020
Later among the works it cites.
Contrastive behavioral similarity embeddings for generalization in reinforcement learning
Agarwal, R., Machado, M. C., Castro, P. S., and Bellemare, M. G · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…