mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization

Understand

Modern deep learning models are over-parameterized, where different optima can result in widely varying generalization performance.

• The Sharpness-Aware Minimization (SAM) technique modifies the fundamental loss function that steers gradient descent methods toward flatter minima, which are believed to exhibit enhanced generalization prowess.
• Our study delves into a specific variant of SAM known as micro-batch SAM (mSAM).
• This variation involves aggregating updates derived from adversarial perturbations across multiple shards (micro-batches) of a mini-batch during training.