Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization

Understand

It is well-known that stochastic gradient noise (SGN) acts as implicit regularization for deep learning and is essentially important for both optimization and generalization of deep networks.

• Some works attempted to artificially simulate SGN by injecting random noise to improve deep learning.
• However, it turned out that the injected simple random noise cannot work as well as SGN, which is anisotropic and parameter-dependent.
• For simulating SGN at low computational costs and without changing the learning rate or batch size, we propose the Positive-Negative Momentum (PNM) approach that is a powerful alternative to conventional Momentum in classic optimizers.