2016

Demystifying ResNet

Li, Sihan, Jiao, Jiantao, Han, Yanjun et al.

Understand

The Residual Network (ResNet), proposed in He et al.

(2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and generalization error.
It was empirically observed in He et al.
(2015) that stacking more layers of residual blocks with shortcut 2 results in smaller training error, while it is not true for shortcut of length 1 or 3.

Exact calculation of the hessian matrix for the multilayer perceptron, 1992
Chris Bishop · 1992
Earlier work this paper cites.
Toeplitz and circulant matrices: A review
Robert M Gray · 2006
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton · 2010
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Andrew M Saxe, James L McClelland, and Surya Ganguli · 2013
Earlier work this paper cites.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio · 2014
Earlier work this paper cites.

Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Cited alongside, same era.
Escaping from saddle points—online stochastic gradient for tensor decomposition
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan · 2015
Cited alongside, same era.
Convolutional, long short-term memory, fully connected deep neural networks
Tara N Sainath, Oriol Vinyals, Andrew Senior, and Haşim Sak · 2015
Cited alongside, same era.
Identity matters in deep learning
Original
Moritz Hardt and Tengyu Ma · 2016
Cited alongside, same era.
Identity mappings in deep residual networks
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Cited in the paper.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun
Cited in the paper.
Training very deep networks
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber
Cited in the paper.
Highway networks
Original
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber
Cited in the paper.

Inception-v4, inception-resnet and the impact of residual connections on learning
Original
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi · 2016
Closest in time.
Resnet in resnet: Generalizing residual architectures
Original
Sasha Targ, Diogo Almeida, and Kevin Lyman · 2016
Closest in time.
Residual networks are exponential ensembles of relatively shallow networks
Original
Andreas Veit, Michael Wilber, and Serge Belongie · 2016
Closest in time.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Closest in time.
Residual networks of residual networks: Multilevel residual networks
Original
Ke Zhang, Miao Sun, Tony X Han, Xingfang Yuan, Liru Guo, and Tao Liu · 2016
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…