Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models

Understand

Fine-tuning large pretrained language models on a limited training corpus usually suffers from poor generalization.

• Prior works show that the recently-proposed sharpness-aware minimization (SAM) optimization method can improve the model generalization.
• However, SAM adds a perturbation to each model parameter equally (but not all parameters contribute equally to the optimization of training), which we argue is sub-optimal and will lead to excessive computation.
• In this paper, we propose a novel optimization procedure, namely FSAM, which introduces a Fisher mask to improve the efficiency and performance of SAM.