Fetching the paper…

Reading the bibliography…

2019

Efficient Computation of Hessian Matrices in TensorFlow

Nilsen, Geir K., Munthe-Kaas, Antonella Z., Skaug, Hans J. et al.

Original
PDF
Post on X

Understand

The Hessian matrix has a number of important applications in a variety of different fields, such as optimzation, image processing and statistics.

In this paper we focus on the practical aspects of efficiently computing Hessian matrices in the context of deep learning using the Python scripting language and the TensorFlow library.
We define a general feed-forward neural network model and show how to efficiently compute two quantities: the cost function's exact Hessian matrix, and the cost function's approximate Hessian matrix, known as the Outer Product of Gradients (OPG) matrix.
Furthermore, as the number of parameters (weights and biases) in deep learning usually is very large, we show how to reduce the quadratic space complexity by an efficient implementation based on approximate eigendecompositions.

Efficient Computation of Hessian Matrices in TensorFlow · Around

Built on

Barak A. Pearlmutter Fast Exact Multiplication by the Hessian
1993
Earlier work this paper cites.
Whitney K. Newey and Daniel McFadden Handbook of Econometrics, Volume 4, Chapter 36 Large sample estimation and hypothesis testing, Pages 2111-2245
1994
Earlier work this paper cites.
, L. N. Trefethen and D. Bau III Numerical Linear Algebra, pp. 243-284
1997
Earlier work this paper cites.

Similar

A. Levy and M. Lindenbaum Sequential Karhunen?Loeve Basis Extraction and its Application to Images
2000
Cited alongside, same era.
Jorge Nocedal and Stephen Wright Numerical Optimization
2000
Cited alongside, same era.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems
2015
Cited alongside, same era.
pyhessian Python Module
Original
Cited in the paper.

Then

Ian Goodfellow Efficient Per-Example Gradient Computations
Original
2015
Later among the works it cites.
H. Cardot and D. Degras Online Principal Component Analysis in High Dimension: Which Algorithm to Choose?
Original
2015
Later among the works it cites.
Ian Goodfellow and Yoshua Bengio and Aaron Courville Deep Learning
2016
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…