A Variant of Gradient Descent Algorithm Based on Gradient Averaging

Understand

In this work, we study an optimizer, Grad-Avg to optimize error functions.

• We establish the convergence of the sequence of iterates of Grad-Avg mathematically to a minimizer (under boundedness assumption).
• We apply Grad-Avg along with some of the popular optimizers on regression as well as classification tasks.
• In regression tasks, it is observed that the behaviour of Grad-Avg is almost identical with Stochastic Gradient Descent (SGD).