2020

Post-Training BatchNorm Recalibration

Shomron, Gil, Weiser, Uri

Understand

We revisit non-blocking simultaneous multithreading (NB-SMT) introduced previously by Shomron and Weiser (2020).

NB-SMT trades accuracy for performance by occasionally "squeezing" more than one thread into a shared multiply-and-accumulate (MAC) unit.
However, the method of accommodating more than one thread in a shared MAC unit may contribute noise to the computations, thereby changing the internal statistics of the model.
We show that substantial model performance can be recouped by post-training recalibration of the batch normalization layers' running mean and running variance statistics, given the presence of NB-SMT.

Built on

Performance estimation of multistreamed, superscalar processors
W. Yamamoto, M. J. Serrano, A. R. Talcott, R. C. Wood, and M. Nemirosky · 1994
Earlier work this paper cites.
Simultaneous multithreading: Maximizing on-chip parallelism
D. M. Tullsen, S. J. Eggers, and H. M. Levy · 1995
Earlier work this paper cites.
Increasing superscalar performance through multistreaming
W. Yamamoto and M. Nemirovsky · 1995
Earlier work this paper cites.
Simultaneous multithreading: A platform for next-generation processors
S. J. Eggers, J. S. Emer, H. M. Levy, J. L. Lo, R. L. Stamm, and D. M. Tullsen · 1997
Earlier work this paper cites.
cuDNN: Efficient primitives for deep learning
Original
S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer · 2014
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
S. Han, J. Pool, J. Tran, and W. Dally · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
ImageNet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Earlier work this paper cites.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.

Similar

Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam · 2017
Cited alongside, same era.
Densely connected convolutional networks
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger · 2017
Cited alongside, same era.
Batch renormalization: Towards reducing minibatch dependence in batch-normalized models
S. Ioffe · 2017
Cited alongside, same era.
Double forward propagation for memorized batch normalization
Y. Guo, Q. Wu, C. Deng, J. Chen, and M. Tan · 2018
Cited alongside, same era.
Quantizing deep convolutional networks for efficient inference: A whitepaper
Original
R. Krishnamoorthi · 2018
Cited alongside, same era.
Post training 4-bit quantization of convolutional networks for rapid-deployment
R. Banner, Y. Nahshan, and D. Soudry · 2019
Cited alongside, same era.
PyTorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala · 2019
Cited alongside, same era.
SMT-SA: Simultaneous multithreading in systolic arrays
G. Shomron, T. Horowitz, and U. Weiser · 2019
Cited alongside, same era.
EvalNorm: Estimating batch normalization statistics for evaluation
S. Singh and A. Shrivastava · 2019
Cited alongside, same era.

Then

Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks
X. Sun, J. Choi, C.-Y. Chen, N. Wang, S. Venkataramani, V. V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan · 2019
Later among the works it cites.
Improving post training neural quantization: Layer-wise calibration and integer programming
Original
I. Hubara, Y. Nahshan, Y. Hanani, R. Banner, and D. Soudry · 2020
Closest in time.
MLPerf inference benchmark
V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, et al · 2020
Closest in time.
Improving robustness against common corruptions by covariate shift adaptation
Original
S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge · 2020
Closest in time.
Robust quantization: One model to rule them all
Original
M. Shkolnik, B. Chmiel, R. Banner, G. Shomron, Y. Nahshan, A. Bronstein, and U. Weiser · 2020
Closest in time.
Non-blocking simultaneous multithreading: Embracing the resiliency of deep neural networks
G. Shomron and U. Weiser · 2020
Closest in time.
Thanks for nothing: Predicting zero-valued activations with lightweight convolutional neural networks
G. Shomron, R. Banner, M. Shkolnik, and U. Weiser · 2020
Closest in time.
Four things everyone should know to improve batch normalization
C. Summers and M. J. Dinneen · 2020
Closest in time.
Calibrated BatchNorm: Improving robustness against noisy weights in neural networks
Original
L.-H. Tsai, S.-C. Chang, Y.-T. Chen, J.-Y. Pan, W. Wei, and D.-C. Juan · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…