Understand
Deep Residual Networks present a premium in performance in comparison to conventional networks of the same depth and are trainable at extreme depths.
- It has recently been shown that Residual Networks behave like ensembles of relatively shallow networks.
- We show that these ensembles are dynamic: while initially the virtual ensemble is mostly at depths lower than half the network's depth, as training progresses, it becomes deeper and deeper.
- The main mechanism that controls the dynamic ensemble behavior is the scaling introduced, e.g., by the Batch Normalization technique.
Built on
Neural networks: tricks of the trade
Genevieve B Orr and Klaus-Robert Müller · 2003
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Complexity of random smooth functions on the high-dimensional sphere
Antonio Auffinger and Gerard Ben Arous · 2013
Earlier work this paper cites.
Random matrices and complexity of spin glasses
Antonio Auffinger, Gérard Ben Arous, and Jiří Černý · 2013
Earlier work this paper cites.
Similar
The loss surfaces of multilayer networks
Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Then
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber · 2015
Later among the works it cites.
Identity mappings in deep residual networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Closest in time.
Residual networks behave like ensembles of relatively shallow networks
Andreas Veit, Michael Wilber, and Serge Belongie · 2016
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…