2018

Norm matters: efficient and accurate normalization schemes in deep networks

Hoffer, Elad, Banner, Ron, Golan, Itay et al.

Understand

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications.

However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks.
In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective.
This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments.

Built on

A simple weight decay can improve generalization
Krogh, A. and Hertz, J. A · 1992
Earlier work this paper cites.
Optimal weight decay in a perceptron
Bös, S · 1996
Earlier work this paper cites.
Using weight decay to optimize the generalization ability of a perceptron
Bos, S. and Chug, E · 1996
Earlier work this paper cites.
Probability distributions involving gaussian random variables: A handbook for engineers and scientists
Simon, M. K · 2007
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, J., Hazan, E., and Singer, Y · 2011
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Bahdanau, D., Cho, K., and Bengio, Y · 2014
Earlier work this paper cites.
Training deep neural networks with low precision multiplications
Original
Courbariaux, M., Bengio, Y., and David, J.-P · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
Expectation backpropagation: parameter-free training of multilayer neural networks with continuous or discrete weights
Soudry, D., Hubara, I., and Meir, R · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.

Similar

Normalization propagation: A parametric technique for removing internal covariate shift in deep networks
Arpit, D., Zhou, Y., Kota, B., and Govindaraju, V · 2016
Cited alongside, same era.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Cited alongside, same era.
Recurrent batch normalization
Original
Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., and Courville, A · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Binarized neural networks
Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y · 2016
Cited alongside, same era.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Salimans, T. and Kingma, D. P · 2016
Cited alongside, same era.
Improved techniques for training gans
Salimans, T., Goodfellow, I., Zaremba, W., et al · 2016
Cited alongside, same era.
Instance normalization: The missing ingredient for fast stylization
Original
Ulyanov, D., Vedaldi, A., and Lempitsky, V. S · 2016
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Original
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2016
Cited alongside, same era.
Shifting mean activation towards zero with bipolar activation functions
Original
Eidnes, L. and Nøkland, A · 2017
Cited alongside, same era.
Comparison of batch normalization and weight normalization algorithms for the large-scale image classification
Original
Gitman, I. and Ginsburg, B · 2017
Cited alongside, same era.
Train longer, generalize better: closing the generalization gap in large batch training of neural networks
Hoffer, E., Hubara, I., and Soudry, D · 2017
Cited alongside, same era.

Then

Self-normalizing neural networks
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S · 2017
Later among the works it cites.
Flexpoint: An adaptive numerical format for efficient training of deep neural networks
Köster, U., Webb, T., Wang, X., et al · 2017
Later among the works it cites.
In-place activated batchnorm for memory-optimized training of dnns
Original
Rota Bulò, S., Porzi, L., and Kontschieder, P · 2017
Later among the works it cites.
Don’t decay the learning rate, increase the batch size
Original
Smith, S. L., Kindermans, P.-J., and Le, Q. V · 2017
Later among the works it cites.
L2 regularization versus batch and weight normalization
Original
van Laarhoven, T · 2017
Later among the works it cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., et al · 2017
Later among the works it cites.
Accelerating deep convolutional networks using low-precision and sparsity
Venkatesh, G., Nurvitadhi, E., and Marr, D · 2017
Later among the works it cites.
On the effect of batch normalization and weight normalization in generative adversarial networks
Original
Xiang, S. and Li, H · 2017
Later among the works it cites.
Mixed precision training of convolutional neural networks using integer operations
Original
Das, D., Mellempudi, N., Mudigere, D., et al · 2018
Closest in time.
Mixed precision training
Micikevicius, P., Narang, S., Alben, J., et al · 2018
Closest in time.
The implicit bias of gradient descent on separable data
Soudry, D., Hoffer, E., and Srebro, N · 2018
Closest in time.
Spectral normalization for generative adversarial networks
Takeru Miyato, M. K. Y. Y., Toshiki Kataoka · 2018
Closest in time.
L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
Wu, S., Li, G., Deng, L., et al · 2018
Closest in time.
Group normalization
Original
Wu, Y. and He, K · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…