2022

The Unsurprising Effectiveness of Pre-Trained Vision Models for Control

Parisi, Simone, Rajeswaran, Aravind, Purushwalkam, Senthil et al.

Understand

Recent years have seen the emergence of pre-trained representations as a powerful abstraction for AI applications in computer vision, natural language, and speech.

However, policy learning for control is still dominated by a tabula-rasa learning paradigm, with visuo-motor policies often trained from scratch using data from deployment environments.
In this context, we revisit and study the role of pre-trained visual representations for control, and in particular representations trained on large-scale computer vision datasets.
Through extensive empirical evaluation in diverse control domains (Habitat, DeepMind Control, Adroit, Franka Kitchen), we isolate and study the importance of different representation training methods, data augmentations, and feature hierarchies.

Built on

Reinforcement Learning: An Introduction
Sutton, R. S. and Barto, A. G · 1998
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Abbeel, P. and Ng, A. Y · 2004
Earlier work this paper cites.
Reducing the dimensionality of data with neural networks
Hinton, G. E. and Salakhutdinov, R. R · 2006
Earlier work this paper cites.
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y · 2012
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Girshick, R., Donahue, J., Darrell, T., and Malik, J · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Zeiler, M. D. and Fergus, R · 2014
Earlier work this paper cites.
Unsupervised visual representation learning by context prediction
Doersch, C., Gupta, A., and Efros, A. A · 2015
Earlier work this paper cites.
Fast R-CNN
Girshick, R · 2015
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Fei-Fei, L · 2015
Earlier work this paper cites.
Trust region policy optimization
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P · 2015
Earlier work this paper cites.
OpenAI Gym
Original
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Generative adversarial imitation learning
Ho, J. and Ermon, S · 2016
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2016
Earlier work this paper cites.
Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge
Vinyals, O., Toshev, A., Bengio, S., and Erhan, D · 2016
Earlier work this paper cites.
Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S., and Sheikh, Y · 2017
Earlier work this paper cites.
Mask R-CNN
He, K., Gkioxari, G., Dollár, P., and Girshick, R · 2017
Earlier work this paper cites.
Emergence of locomotion behaviours in rich environments
Original
Heess, N. M. O., Dhruva, T., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., Riedmiller, M. A., and Silver, D · 2017
Earlier work this paper cites.

Similar

Feature pyramid networks for object detection
Lin, T., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S · 2017
Cited alongside, same era.
Towards generalization and simplicity in continuous control
Rajeswaran, A., Lowrey, K., Todorov, E. V., and Kakade, S. M · 2017
Cited alongside, same era.
Grad-CAM: Visual explanations from deep networks via gradient-based localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D · 2017
Cited alongside, same era.
Revisiting unreasonable effectiveness of data in deep learning era
Sun, C., Shrivastava, A., Singh, S., and Gupta, A · 2017
Cited alongside, same era.
Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning
Tieleman, T. and Hinton, G · 2017
Cited alongside, same era.
Places: A 10 million image database for scene recognition
Original
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A · 2017
Cited alongside, same era.
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al · 2018
Cited alongside, same era.
Deep quaternion networks
Gaudet, C. J. and Maida, A · 2018
Cited alongside, same era.
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., and Satoh, Y · 2018
Cited alongside, same era.
DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
Peng, X. B., Abbeel, P., Levine, S., and van de Panne, M · 2018
Cited alongside, same era.
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S · 2018
Cited alongside, same era.
DeepMind Control Suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T. P., and Riedmiller, M. A · 2018
Cited alongside, same era.

Then

A Simple Framework for Contrastive Learning of Visual Representations
Original
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E · 2020
Later among the works it cites.
D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Original
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S · 2020
Later among the works it cites.
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T. P., Ba, J., and Norouzi, M · 2020
Later among the works it cites.
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B · 2020
Later among the works it cites.
Reinforcement learning with augmented data
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A · 2020
Later among the works it cites.
Learning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., Schneider, J., Sidor, S., Tobin, J., Welinder, P., Weng, L., and Zaremba, W · 2020
Later among the works it cites.
Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases
Purushwalkam, S. and Gupta, A · 2020
Later among the works it cites.
CURL: Contrastive Unsupervised Representations for Reinforcement Learning
Srinivas, A., Laskin, M., and Abbeel, P · 2020
Later among the works it cites.
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
Wijmans, E., Kadian, A., Morcos, A. S., Lee, S., Essa, I., Parikh, D., Savva, M., and Batra, D · 2020
Later among the works it cites.
Learning to see before learning to act: Visual pre-training for manipulation
Yen-Chen, L., Zeng, A., Song, S., Isola, P., and Lin, T · 2020
Later among the works it cites.
Self-supervised pretraining of visual features in the wild
Original
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al · 2021
Later among the works it cites.
Simple but Effective: CLIP Embeddings for Embodied AI
Original
Khandelwal, A., Weihs, L., Mottaghi, R., and Kembhavi, A · 2021
Later among the works it cites.
Interesting Object, Curious Agent: Learning Task-Agnostic Exploration
Parisi, S., Dean, V., Pathak, D., and Gupta, A · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Later among the works it cites.
Visual adversarial imitation learning using variational models
Rafailov, R., Yu, T., Rajeswaran, A., and Finn, C · 2021
Later among the works it cites.
RRL: ResNet as representation for Reinforcement Learning
Shah, R. and Kumar, V · 2021
Later among the works it cites.
Reinforcement Learning with Latent Flow
Shang, W., Wang, X., Srinivas, A., Rajeswaran, A., Gao, Y., Abbeel, P., and Laskin, M · 2021
Later among the works it cites.
Decoupling representation learning from reinforcement learning
Stooke, A., Lee, K., Abbeel, P., and Laskin, M · 2021
Later among the works it cites.
Visual room rearrangement
Weihs, L., Deitke, M., Kembhavi, A., and Mottaghi, R · 2021
Later among the works it cites.
Learning invariant representations for reinforcement learning without reconstruction
Zhang, A., McAllister, R., Calandra, R., Gal, Y., and Levine, S · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…