Sharpness-Aware Minimization for Efficiently Improving Generalization

Understand

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability.

• Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality.
• Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness.
• In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently.