Training data-efficient image transformers & distillation through attention

Understand

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification.

• However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption.
• In this work, we produce a competitive convolution-free transformer by training on Imagenet only.
• We train them on a single computer in less than 3 days.