Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding

Understand

Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models.

• Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks.
• As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting.
• In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks.