Convolutional Bypasses Are Better Vision Transformer Adapters

Understand

The pretrain-then-finetune paradigm has been widely adopted in computer vision.

• But as the size of Vision Transformer (ViT) grows exponentially, the full finetuning becomes prohibitive in view of the heavier storage overhead.
• Motivated by parameter-efficient transfer learning (PETL) on language transformers, recent studies attempt to insert lightweight adaptation modules (e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune these modules while the pretrained weights are frozen.
• However, these modules were originally proposed to finetune language models and did not take into account the prior knowledge specifically for visual tasks.