Revisiting inverse Hessian vector products for calculating influence functions

Understand

Influence functions are a popular tool for attributing a model's output to training data.

• The traditional approach relies on the calculation of inverse Hessian-vector products (iHVP), but the classical solver "Linear time Stochastic Second-order Algorithm" (LiSSA, Agarwal et al.
• (2017)) is often deemed impractical for large models due to expensive computation and hyperparameter tuning.
• We show that the three hyperparameters -- the scaling factor, the batch size, and the number of steps -- can be chosen depending on the spectral properties of the Hessian, particularly its trace and largest eigenvalue.