Entropic Risk-Averse Generalized Momentum Methods

Understand

In the context of first-order algorithms subject to random gradient noise, we study the trade-offs between the convergence rate (which quantifies how fast the initial conditions are forgotten) and the "risk" of suboptimality, i.e.

• deviations from the expected suboptimality.
• We focus on a general class of momentum methods (GMM) which recover popular methods such as gradient descent (GD), accelerated gradient descent (AGD), and heavy-ball (HB) method as special cases depending on the choice of GMM parameters.
• We use well-known risk measures "entropic risk" and "entropic value at risk" to quantify the risk of suboptimality.