Fetching the paper…

Reading the bibliography…

2019

A Reparameterization-Invariant Flatness Measure for Deep Neural Networks

Petzka, Henning, Adilova, Linara, Kamp, Michael et al.

Original
PDF
Post on X

Understand

The performance of deep neural networks is often attributed to their automated, task-related feature construction.

It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.
Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.
For several flatness measures, this correlation has been empirically validated.

A Reparameterization-Invariant Flatness Measure for Deep Neural Networks · Around

Built on

Simplifying neural nets by discovering flat minima
S. Hochreiter and J. Schmidhuber · 1995
Earlier work this paper cites.
Flat minima
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
On large-batch training for deep learning: Generalization gap and sharp minima
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang · 2016
Earlier work this paper cites.

Similar

Sharp minima can generalize for deep nets
L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio · 2017
Cited alongside, same era.
Sensitivity and generalization in neural networks: an empirical study
R. Novak, Y. Bahri, D. A. Abolafia, J. Pennington, and J. Sohl-Dickstein · 2018
Cited alongside, same era.
Identifying generalization properties in neural networks
Original
H. Wang, N. S. Keskar, C. Xiong, and R. Socher · 2018
Cited alongside, same era.
Lenet-5, convolutional neural networks
Y. LeCun et al
Cited in the paper.

Then

An investigation into neural net optimization via hessian eigenvalue density
Original
B. Ghorbani, S. Krishnan, and Y. Xiao · 2019
Closest in time.
A scale invariant flatness measure for deep network minima
Original
A. Rangamani, N. H. Nguyen, A. Kumar, D. T. Phan, S. H. Chin, and T. D. Tran · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…