HesScale: Scalable Computation of Hessian Diagonals

Understand

Second-order optimization uses curvature information about the objective function, which can help in faster convergence.

• However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a scalable way.
• The absence of efficient ways of computation drove the most widely used methods to focus on first-order approximations that do not capture the curvature information.
• In this paper, we develop HesScale, a scalable approach to approximating the diagonal of the Hessian matrix, to incorporate second-order information in a computationally efficient manner.