2020

Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization

Tsai, Li-Huang, Chang, Shih-Chieh, Chen, Yu-Ting et al.

Understand

Analog computing hardwares, such as Processing-in-memory (PIM) accelerators, have gradually received more attention for accelerating the neural network computations.

However, PIM accelerators often suffer from intrinsic noise in the physical components, making it challenging for neural network models to achieve the same performance as on the digital hardware.
Previous works in mitigating intrinsic noise assumed the knowledge of the noise model, and retraining the neural networks accordingly was required.
In this paper, we propose a noise-agnostic method to achieve robust neural network performance against any noise setting.

Built on

Nitish Srivastava and Geoffrey Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov, ”Dropout: A Simple Way to Prevent Neural Networks from Overfitting”, in Journal of Machine Learning Research
1958
Earlier work this paper cites.
S. Yu, X. Guan and H.-S. Philip Wong, ”On the stochastic nature of resistive switching in metal oxide RRAM: physical modeling, Monte Carlo simulation, and experimental characterization”, in Proceedings of the IEDM
2011
Earlier work this paper cites.
S. R. Lee and Y. Kim and M. Chang and K. M. Kim and C. B. Lee and J. H. Hur and G. Park and D. Lee and M. Lee and C. J. Kim and U. Chung and I. Yoo and K. Kim, ”Multi-level switching of triple-layered TaOx RRAM with excellent reliability for storage class memory”, in Symposium on VLSI Technology
2012
Earlier work this paper cites.
Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E, ”ImageNet classification with deep convolutional neural networks”, in Conference on Neural Information Processing Systems
2012
Earlier work this paper cites.
Everingham, M. and Van Gool, L. and Williams, C. K. I. and Winn, J. and Zisserman, A., ”The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results”, 2012
2012
Earlier work this paper cites.
Tsung-Yi Lin and Michael Maire et al., ”Microsoft COCO: Common Objects in Context”, preprint arXiv:1405.0312
Original
2014
Earlier work this paper cites.
Sergey Ioffe and Christian Szegedy, ”Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, in Proceedings of Machine Learning Research
2015
Earlier work this paper cites.
Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun, ”Deep Residual Learning for Image Recognition”, preprint arXiv:1512.03385
Original
2015
Earlier work this paper cites.
Jonathan Long and Evan Shelhamer and Trevor Darrell, ”Fully Convolutional Networks for Semantic Segmentation”, in CVPR
2015
Earlier work this paper cites.
A. Shafiee and A. Nag and N. Muralimanohar and R. Balasubramonian and J. P. Strachan and M. Hu and R. S. Williams and V. Srikumar, ”ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars” in Annual International Symposium on Computer Architecture
2016
Earlier work this paper cites.
Sergey Zagoruyko and Nikos Komodakis, ”Wide Residual Networks”, in BMVC
2016
Earlier work this paper cites.

Similar

W. Wu and H. Wu and B. Gao and N. Deng and S. Yu and H. Qian, ”Improving Analog Switching in HfOx-Based Resistive Memory With a Thermal Enhanced Layer”, in IEEE Electron Device Letters
2017
Cited alongside, same era.
S. R. Nandakumar, I. Boybat, M. Le Gallo, A. Sebastian, B. Rajendran and E. Eleftheriou, ”Supervised learning in spiking neural networks with MLC PCM synapses”, in Annual Device Research Conference
2017
Cited alongside, same era.
Minghai Qin and Dejan Vucinic, ”Training Recurrent Neural Networks against Noisy Computations during Inference”, in Asilomar Conference on Signals, Systems, and Computers
2018
Cited alongside, same era.
Yong Guo and Qingyao Wu and Chaorui Deng and Jian Jhen Chen and Mingkui Tan, ”Double Forward Propagation for Memorized Batch Normalization”, in AAAI
2018
Cited alongside, same era.
Changhyuck Sung, Seokjae Lim, Hyungjun Kim, Taesu Kim, Kibong Moon, Jeonghwan Song, Jae-Joon Kim and Hyunsang Hwang, ”Effect of conductance linearity and multi-level cell characteristics of TaOx-based synapse device on pattern recognition accuracy of neuromorphic system”, in Nanotechnology
2018
Cited alongside, same era.
S. Lim and C. Sung and H. Kim and T. Kim and J. Song and J. Kim and H. Hwang, ”Improved Synapse Device With MLC and Conductance Linearity Using Quantized Conduction for Neuromorphic Systems”, in IEEE Electron Device Letters
2018
Cited alongside, same era.
Joseph Redmon and Ali Farhadi, ”YOLOv3: An Incremental Improvement”, preprint arXiv:1804.02767
Original
2018
Cited alongside, same era.
Mark Sandler and Andrew Howard and Menglong Zhu and Andrey Zhmoginov and Liang-Chieh Chen, ”MobileNetV2: Inverted Residuals and Linear Bottlenecks”, in CVPR
2018
Cited alongside, same era.
S. Angizi and Z. He and D. Reis and X. S. Hu and W. Tsai and S. J. Lin and D. Fan, ”Accelerating Deep Neural Networks in Processing-in-Memory Platforms: Analog or Digital Approach?” in Computer Society Annual Symposium on VLSI
2019
Cited alongside, same era.
W. Haensch and T. Gokmen and R. Puri, ”The Next Generation of Deep Learning Hardware: Analog Computing”, in Proceedings of the IEEE
2019
Cited alongside, same era.

Then

Michael Klachko and Mohammad Reza Mahmoodi and Dmitri B. Strukov, ”Improving Noise Tolerance of Mixed-Signal Neural Networks”, in IJCNN
2019
Later among the works it cites.
Saurabh Singh and Abhinav Shrivastava, ”EvalNorm: Estimating Batch Normalization Statistics for Evaluation”, in ICCV
2019
Later among the works it cites.
Adarsha Balaji and Shihao Song and Anup Das and Nikil Dutt and Jeff Krichmar and Nagarajan Kandasamy and Francky Catthoor, ”A Framework to Explore Workload-Specific Performance and Lifetime Trade-offs in Neuromorphic Computing”, in IEEE Computer Architecture Letters
2019
Later among the works it cites.
Ruiqi Gao and Tianle Cai and Haochuan Li and Liwei Wang and Cho-Jui Hsieh and Jason D. Lee, ”Convergence of Adversarial Training in Overparametrized Neural Networks”, in NeurIPS
2019
Later among the works it cites.
Y. Zhu and G. L. Zhang and T. Wang and B. Li and Y. Shi and T. Ho and U. Schlichtmann, ”Statistical Training for Neuromorphic Computing using Memristor-based Crossbars Considering Process Variations and Noise”, in DATE
2020
Closest in time.
Hyeongsu Kim, Jong-Ho Bae, SuhwanLim, Sung-Tae Lee, Young-Tak Seo, Dongseok Kwon, Byung-Gook Park and Jong-Ho Lee, ”Efficient precise weight tuning protocol considering variation of the synaptic devices and target accuracy”, in Neurocomputing
2020
Closest in time.
Chuteng Zhou and Prad Kadambi and Matthew Mattina and Paul N. Whatmough, ”Noisy Machines: Understanding Noisy Neural Networks and Enhancing Robustness to Analog Hardware Errors Using Distillation” preprint arXiv: 2001.04974
2020
Closest in time.
Cihang Xie and Mingxing Tan and Boqing Gong and Jiang Wang and Alan Yuille and Quoc V. Le, ”Adversarial Examples Improve Image Recognition”, in CVPR
2020
Closest in time.
Jonathan Frankle and David J. Schwab and Ari S. Morcos, ”Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs”, preprint arXiv: 2003.00152
2020
Closest in time.
Cecilia Summers and Michael J. Dinneen, ”Four Things Everyone Should Know to Improve Batch Normalization”, in ICLR
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…