BackPACK: Packing more into backprop

Understand

Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient.

• Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in theory, be computed efficiently, and at the same time as the gradient.
• While these quantities are of great interest to researchers and practitioners, current deep-learning software does not support their automatic calculation.
• Manually implementing them is burdensome, inefficient if done naively, and the resulting code is rarely shared.