Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning

Understand

The Vision Transformer architecture has shown to be competitive in the computer vision (CV) space where it has dethroned convolution-based networks in several benchmarks.

• Nevertheless, convolutional neural networks (CNN) remain the preferential architecture for the representation module in reinforcement learning.
• In this work, we study pretraining a Vision Transformer using several state-of-the-art self-supervised methods and assess the quality of the learned representations.
• To show the importance of the temporal dimension in this context we propose an extension of VICReg to better capture temporal relations between observations by adding a temporal order verification task.