Benchmarking Detection Transfer Learning with Vision Transformers

Understand

Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed.

• The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive.
• These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models.
• In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN.