Fetching the paper…

Reading the bibliography…

2019

A Sober Look at Neural Network Initializations

Steinwart, Ingo

Original
PDF
Post on X

Understand

Initializing the weights and the biases is a key part of the training process of a neural network.

Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature.
In this paper we discuss some consequences of commonly used initialization strategies for vanilla DNNs with ReLU activations.
Based on these insights we then develop an alternative initialization strategy.

A Sober Look at Neural Network Initializations · Around

Built on

On the distribution of the quotient of two chance variables
J. H. Curtiss · 1941
Earlier work this paper cites.
Some elementary inequalities relating to the gamma and incomplete gamma function
W. Gautschi · 1959
Earlier work this paper cites.
Some extensions of W. Gautschi’s inequalities for the gamma function
D. Kershaw · 1983
Earlier work this paper cites.
Non-Uniform Random Variate Generation
L. Devroye · 1986
Earlier work this paper cites.
Inequalities for the incomplete gamma function
P. Natalini and B. Palumbo · 2000
Earlier work this paper cites.
Convex Analysis: Theory and Applications
G. G. Magaril-Il’yaev and V. M. Tikhomirov · 2003
Earlier work this paper cites.

Similar

Support Vector Machines
I. Steinwart and A. Christmann · 2008
Cited alongside, same era.
Uniform bounds for the complementary incomplete gamma function
J. M. Borwein and O-Y. Chan · 2009
Cited alongside, same era.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Cited alongside, same era.
Superconcentration and Related Topics
S. Chatterjee · 2014
Cited alongside, same era.

Then

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Later among the works it cites.
Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Later among the works it cites.
Analysis 3
O. Forster · 2017
Later among the works it cites.
Self-normalizing neural networks
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter · 2017
Later among the works it cites.
High-Dimensional Probability
Original
R. Vershynin · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…