Fetching the paper…

Reading the bibliography…

2016

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

Sagun, Levent, Bottou, Leon, LeCun, Yann

Original
PDF
Post on X

Understand

We look at the eigenvalues of the Hessian of a loss function before and after training.

The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero.
We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond · Around

Built on

Fast exact multiplication by the hessian
Barak A Pearlmutter · 1994
Earlier work this paper cites.
Almost all learning machines are singular
Sumio Watanabe · 2007
Earlier work this paper cites.
On optimization methods for deep learning
Jiquan Ngiam, Adam Coates, Ahbik Lahiri, Bobby Prochnow, Quoc V Le, and Andrew Y Ng · 2011
Earlier work this paper cites.
Efficient backprop
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller · 2012
Earlier work this paper cites.

Similar

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio · 2014
Cited alongside, same era.
Explorations on high dimensional landscapes
Original
Levent Sagun, V Uğur Güney, Gérard Ben Arous, and Yann LeCun · 2014
Cited alongside, same era.
Train faster, generalize better: Stability of stochastic gradient descent
Original
Moritz Hardt, Benjamin Recht, and Yoram Singer · 2015
Cited alongside, same era.
Universality in halting time and its applications in optimization
Original
Levent Sagun, Thomas Trogdon, and Yann LeCun · 2015
Cited alongside, same era.

Then

Topology and geometry of deep rectified network optimization landscapes
Original
C Daniel Freeman and Joan Bruna · 2016
Closest in time.
Gradient descent converges to minimizers
Jason D Lee, Max Simchowitz, Michael I Jordan, and Benjamin Recht · 2016
Closest in time.
Gradient descent only converges to minimizers: Non-isolated critical points and invariant regions
Original
Ioannis Panageas and Georgios Piliouras · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…