Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Points

Understand

The note considers normalized gradient descent (NGD), a natural modification of classical gradient descent (GD) in optimization problems.

• A serious shortcoming of GD in non-convex problems is that GD may take arbitrarily long to escape from the neighborhood of a saddle point.
• This issue can make the convergence of GD arbitrarily slow, particularly in high-dimensional non-convex problems where the relative number of saddle points is often large.
• The paper focuses on continuous-time descent.