A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models

Understand

Finetuning language models (LMs) is crucial for adapting the models to downstream data and tasks.

• However, full finetuning is usually costly.
• Existing work, such as parameter-efficient finetuning (PEFT), often focuses on \textit{how to finetune} but neglects the issue of \textit{where to finetune}.
• As a pioneering work on reducing the cost of backpropagation (at the layer level) by answering where to finetune, we conduct a semantic analysis of the LM inference process.