Understand
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks.
- Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs).
- Also, BN cannot be used with batch-size 1 during training.
- We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation.
Built on
Independent component analysis: algorithms and applications
Hyvärinen, Aapo and Oja, Erkki · 2000
Earlier work this paper cites.
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, Hidetoshi · 2000
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, Alex · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, Xavier and Bengio, Yoshua · 2010
Earlier work this paper cites.
Sparse representation for computer vision and pattern recognition
Wright, John, Ma, Yi, Mairal, Julien, Sapiro, Guillermo, Huang, Thomas S, and Yan, Shuicheng · 2010
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y · 2011
Earlier work this paper cites.
Similar
Efficient backprop
LeCun, Yann A, Bottou, Léon, Orr, Genevieve B, and Müller, Klaus-Robert · 2012
Cited alongside, same era.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tieleman, Tijmen and Hinton, Geoffrey · 2012
Cited alongside, same era.
Maxout networks
Goodfellow, Ian J., Warde-farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua · 2013
Cited alongside, same era.
Makhzani, Alireza and Frey, Brendan · 2013
Cited alongside, same era.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, Andrew M, McClelland, James L, and Ganguli, Surya · 2013
Cited alongside, same era.
Network in network
Lin, Min, Chen, Qiang, and Yan, Shuicheng · 2014
Cited alongside, same era.
Then
Learning activation functions to improve deep neural networks
Agostinelli, Forest, Hoffman, Matthew, Sadowski, Peter, and Baldi, Pierre · 2015
Later among the works it cites.
Natural neural networks
Desjardins, Guillaume, Simonyan, Karen, Pascanu, Razvan, et al · 2015
Later among the works it cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian · 2015
Later among the works it cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, Sergey and Szegedy, Christian · 2015
Later among the works it cites.
Deeply-supervised nets
Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen · 2015
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…