WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Understand

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems.

• Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context.
• Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian.
• Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework.