Fetching the paper…

Reading the bibliography…

2020

Generalization in Reinforcement Learning by Soft Data Augmentation

Hansen, Nicklas, Wang, Xiaolong

Original
PDF
Post on X

Understand

Extensive efforts have been made to improve the generalization ability of Reinforcement Learning (RL) methods via domain randomization and data augmentation.

However, as more factors of variation are introduced during training, optimization becomes increasingly challenging, and empirically may result in lower sample efficiency and unstable training.
Instead of learning policies directly from augmented data, we propose SOft Data Augmentation (SODA), a method that decouples augmentation from policy learning.
Specifically, SODA imposes a soft constraint on the encoder that aims to maximize the mutual information between latent representations of augmented and non-augmented data, while the RL optimization process uses strictly non-augmented data.

Generalization in Reinforcement Learning by Soft Data Augmentation · Around

Built on

A. Stooke, K. Lee, P. Abbeel, and M. Laskin, “Decoupling representation learning from reinforcement learning,” arXiv:2004.14990
Original
2004
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602 , 2013
Original
2013
Earlier work this paper cites.
X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 2794–2802
2015
Earlier work this paper cites.
C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 1422–1430
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ArXiv , vol. abs/1502.03167, 2015
Original
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR , vol. abs/1412.6980, 2015
Original
2015
Earlier work this paper cites.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” The Journal of Machine Learning Research , vol. 17, no. 1, pp. 1334–1373, 2016
2016
Earlier work this paper cites.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2536–2544
2016
Earlier work this paper cites.
M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in European Conference on Computer Vision . Springer, 2016, pp. 69–84
2016
Earlier work this paper cites.
R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in European conference on computer vision . Springer, 2016, pp. 649–666
2016
Earlier work this paper cites.
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” CoRR , vol. abs/1509.02971, 2016
Original
2016
Earlier work this paper cites.
Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in ICRA . IEEE, 2017, pp. 3357–3364
2017
Earlier work this paper cites.
L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel, “Asymmetric actor critic for image-based robot learning,” arXiv preprint arXiv:1710.06542 , 2017
Original
2017
Earlier work this paper cites.
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Sep 2017
2017
Earlier work this paper cites.
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017
Original
2017
Earlier work this paper cites.
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual reinforcement learning with imagined goals,” in Advances in Neural Information Processing Systems , 2018, pp. 9191–9200
2018
Earlier work this paper cites.

Similar

D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” ArXiv , vol. abs/1806.10293, 2018
Original
2018
Cited alongside, same era.
K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman, “Quantifying generalization in reinforcement learning,” 2018
2018
Cited alongside, same era.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” 2018 IEEE International Conference on Robotics and Automation (ICRA) , May 2018
2018
Cited alongside, same era.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller, “DeepMind control suite,” https://arxiv.org/abs/1801.00690, DeepMind, Tech. Rep., Jan. 2018
Original
2018
Cited alongside, same era.

Then

T. Carr, M. Chli, and G. Vogiatzis, “Domain adaptation for reinforcement learning on the atari,” in AAMAS , 2019
2019
Later among the works it cites.
F. Ramos, R. Possas, and D. Fox, “Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators,” Robotics: Science and Systems XV , Jun 2019
2019
Later among the works it cites.
M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas, “Reinforcement learning with augmented data,” arXiv preprint arXiv:2004.14990 , 2020
Original
2020
Closest in time.
R. Raileanu, M. Goldstein, D. Yarats, I. Kostrikov, and R. Fergus, “Automatic data augmentation for generalization in deep reinforcement learning,” ArXiv , vol. abs/2006.12862, 2020
Original
2020
Closest in time.
A. Srinivas, M. Laskin, and P. Abbeel, “Curl: Contrastive unsupervised representations for reinforcement learning,” arXiv preprint arXiv:2004.04136 , 2020

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 3733–3742

2018

Cited alongside, same era.

A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2018

2018

Cited alongside, same era.

C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song, “Assessing generalization in deep reinforcement learning,” 2018

2018

Cited alongside, same era.

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” 2018

2018

Cited alongside, same era.

W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi, “Visual semantic navigation using scene priors,” in ICLR , 2019

2019

Cited alongside, same era.

S. Gamrian and Y. Goldberg, “Transfer learning for related reinforcement learning tasks via image-to-image translation,” ArXiv , vol. abs/1806.07377, 2019

2019

Cited alongside, same era.

X. Song, Y. Jiang, S. Tu, Y. Du, and B. Neyshabur, “Observational overfitting in reinforcement learning,” 2019

2019

Cited alongside, same era.

K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, “Leveraging procedural generation to benchmark reinforcement learning,” ArXiv , vol. abs/1912.01588, 2019

2019

Cited alongside, same era.

2020

Closest in time.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” 2020

2020

Closest in time.

K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, “Momentum contrast for unsupervised visual representation learning,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 9726–9735, 2020

2020

Closest in time.

I. Misra and L. van der Maaten, “Self-supervised learning of pretext-invariant representations,” in CVPR , 2020

2020

Closest in time.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” ArXiv , vol. abs/2006.07733, 2020

2020

Closest in time.

M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman, “Data-efficient reinforcement learning with momentum predictive representations,” arXiv preprint arXiv:2007.05929 , 2020

2020

Closest in time.

X. Chen, A. Ghadirzadeh, M. Björkman, and P. Jensfelt, “Adversarial feature training for generalizable robotic visuomotor control,” 2020 IEEE International Conference on Robotics and Automation (ICRA) , pp. 1142–1148, 2020

2020

Closest in time.

R. Julian, B. Swanson, G. Sukhatme, S. Levine, C. Finn, and K. Hausman, “Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning,” arXiv: Learning , 2020

2020

Closest in time.

I. Kostrikov, D. Yarats, and R. Fergus, “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,” 2020

2020

Closest in time.

N. Hansen, Y. Sun, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang, “Self-supervised policy adaptation during deployment,” 2020

2020

Closest in time.

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning,” ArXiv , vol. abs/2005.10243, 2020

2020

Closest in time.