SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters

Understand

Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tuning.

• Although computationally efficient, the recent Adapters often increase parameters (e.g.
• bottleneck dimension) for matching the performance of full model fine-tuning, which we argue goes against their original intention.
• In this work, we re-examine the parameter-efficiency of Adapters through the lens of network pruning (we name such plug-in concept as \texttt{SparseAdapter}) and find that SparseAdapter can achieve comparable or better performance than standard Adapters when the sparse ratio reaches up to 80\%.