Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning

Understand

While transferring a pretrained language model, common approaches conventionally attach their task-specific classifiers to the top layer and adapt all the pretrained layers.

• We investigate whether one could make a task-specific selection on which subset of the layers to adapt and where to place the classifier.
• The goal is to reduce the computation cost of transfer learning methods (e.g.
• fine-tuning or adapter-tuning) without sacrificing its performance.