Local Saddle Point Optimization: A Curvature Exploitation Approach

Understand

Gradient-based optimization methods are the most popular choice for finding local optima for classical minimization and saddle point problems.

• Here, we highlight a systemic issue of gradient dynamics that arise for saddle point problems, namely the presence of undesired stable stationary points that are no local optima.
• We propose a novel optimization approach that exploits curvature information in order to escape from these undesired stationary points.
• We prove that different optimization methods, including gradient method and Adagrad, equipped with curvature exploitation can escape non-optimal stationary points.