FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer

Understand

Recent work has explored the potential to adapt a pre-trained vision transformer (ViT) by updating only a few parameters so as to improve storage efficiency, called parameter-efficient transfer learning (PETL).

• Current PETL methods have shown that by tuning only 0.5% of the parameters, ViT can be adapted to downstream tasks with even better performance than full fine-tuning.
• In this paper, we aim to further promote the efficiency of PETL to meet the extreme storage constraint in real-world applications.
• To this end, we propose a tensorization-decomposition framework to store the weight increments, in which the weights of each ViT are tensorized into a single 3D tensor, and their increments are then decomposed into lightweight factors.