Fetching the paper…

Reading the bibliography…

2019

How Does Batch Normalization Help Binary Training?

Sari, Eyyüb, Belbahri, Mouloud, Nia, Vahid Partovi

Original
PDF
Post on X

Understand

Binary Neural Networks (BNNs) are difficult to train, and suffer from drop of accuracy.

It appears in practice that BNNs fail to train in the absence of Batch Normalization (BatchNorm) layer.
We find the main role of BatchNorm is to avoid exploding gradients in the case of BNNs.
This finding suggests that the common initialization methods developed for full-precision networks are irrelevant to BNNs.

How Does Batch Normalization Help Binary Training? · Around

Built on

Learning multiple layers of features from tiny images
Krizhevsky, A. 2009 · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Y. Bengio 2010 · 2010
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and J. Ba 2014 · 2014
Earlier work this paper cites.
Deep residual learning for image recognition
Original
He, K., X. Zhang, S. Ren, and J. Sun 2015 · 2015
Earlier work this paper cites.

Similar

Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S. and C. Szegedy 2015 · 2015
Cited alongside, same era.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Howard, A. G., M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam 2017 · 2017
Cited alongside, same era.
Communication-efficient learning of deep networks from decentralized data
McMahan, H. B., E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas 2017 · 2017
Cited alongside, same era.
Quantized neural networks: Training neural networks with low precision weights and activations
Hubara, I., M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio 2018 · 2018
Cited alongside, same era.

Then

How does batch normalization help optimization?
Santurkar, S., D. Tsipras, A. Ilyas, and A. Madry 2018 · 2018
Later among the works it cites.
A systematic study of binary neural networks’ optimisation
Alizadeh, M., J. Fernández-Marqués, N. D. Lane, and Y. Gal 2019 · 2019
Closest in time.
Learning recurrent binary/ternary weights
Ardakani, A., Z. Ji, S. C. Smithson, B. H. Meyer, and W. J. Gross 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…