Perturbated Gradients Updating within Unit Space for Deep Learning

Understand

In deep learning, optimization plays a vital role.

• By focusing on image classification, this work investigates the pros and cons of the widely used optimizers, and proposes a new optimizer: Perturbated Unit Gradient Descent (PUGD) algorithm with extending normalized gradient operation in tensor within perturbation to update in unit space.
• Via a set of experiments and analyses, we show that PUGD is locally bounded updating, which means the updating from time to time is controlled.
• On the other hand, PUGD can push models to a flat minimum, where the error remains approximately constant, not only because of the nature of avoiding stationary points in gradient normalization but also by scanning sharpness in the unit ball.