2020

Sparse Weight Activation Training

Raihan, Md Aamir, Aamodt, Tor M.

Understand

Neural network training is computationally and memory intensive.

Sparse training can reduce the burden on emerging hardware platforms designed to accelerate sparse computations, but it can affect network convergence.
In this work, we propose a novel CNN training algorithm Sparse Weight Activation Training (SWAT).
SWAT is more computation and memory-efficient than conventional training.

Built on

Algorithm 65: Find
C. A. R. Hoare · 1961
Earlier work this paper cites.
Validity of the single processor approach to achieving large scale computing capabilities
Gene M Amdahl · 1967
Earlier work this paper cites.
Time bounds for selection
Manuel Blum, Robert W. Floyd, Vaughan Pratt, Ronald L. Rivest, and Robert E. Tarjan · 1973
Earlier work this paper cites.
Optimal brain damage
Yann LeCun, John S Denker, and Sara A Solla · 1990
Earlier work this paper cites.
Introspective sorting and selection algorithms
David R. Musser · 1997
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky et al · 2009
Earlier work this paper cites.
Efficient approximate top-k query algorithm using cube index
Dongqu Chen, Guang-Zhong Sun, and Neil Zhenqiang Gong · 2011
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
Fast k-selection algorithms for graphics processing units
Tolu Alabi, Jeffrey D Blanchard, Bradley Gordon, and Russel Steinbach · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Deep neural networks for object detection
Christian Szegedy, Alexander Toshev, and Dumitru Erhan · 2013
Earlier work this paper cites.
Dadiannao: A machine-learning supercomputer
Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al · 2014
Earlier work this paper cites.
Learning a deep convolutional network for image super-resolution
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Original
Song Han, Huizi Mao, and William J Dally · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Earlier work this paper cites.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Earlier work this paper cites.
Cnvlutin: Ineffectual-neuron-free deep neural network computing
Jorge Albericio, Patrick Judd, Tayler Hetherington, Tor Aamodt, Natalie Enright Jerger, and Andreas Moshovos · 2016
Earlier work this paper cites.
A guide to convolution arithmetic for deep learning
Original
Vincent Dumoulin and Francesco Visin · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Pruning filters for efficient convnets
Original
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf · 2016
Earlier work this paper cites.

Similar

Pruning convolutional neural networks for resource efficient inference
Original
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz · 2016
Cited alongside, same era.
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna · 2016
Cited alongside, same era.
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Cited alongside, same era.
Quantized convolutional neural networks for mobile devices
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng · 2016
Cited alongside, same era.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Cambricon-x: An accelerator for sparse neural networks
Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li, Qi Guo, Tianshi Chen, and Yunji Chen · 2016
Cited alongside, same era.
Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients
Original
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou · 2016
Cited alongside, same era.
Channel pruning for accelerating very deep neural networks
Yihui He, Xiangyu Zhang, and Jian Sun · 2017
Cited alongside, same era.
Densely connected convolutional networks
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger · 2017
Cited alongside, same era.
In-datacenter performance analysis of a tensor processing unit
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang · 2017
Cited alongside, same era.
Learning sparse neural networks through l _ 0 l\_0 regularization
Original
Christos Louizos, Max Welling, and Diederik P Kingma · 2017
Cited alongside, same era.

Then

Mixed precision training of convolutional neural networks using integer operations
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradeep Dubey, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, and Vadim Pirogov · 2018
Later among the works it cites.
Full deep neural network training on a pruned weight budget
Original
Maximilian Golub, Guy Lemieux, and Mieszko Lis · 2018
Later among the works it cites.
Soft filter pruning for accelerating deep convolutional neural networks
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang · 2018
Later among the works it cites.
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta · 2018
Later among the works it cites.
Compressing dma engine: Leveraging activation sparsity for training deep neural networks
Minsoo Rhu, Mike O’Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Keckler · 2018
Later among the works it cites.
Efficient top-k query processing on massively parallel hardware
Anil Shanbhag, Holger Pirk, and Samuel Madden · 2018
Later among the works it cites.
Training deep neural networks with 8-bit floating point numbers
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan · 2018
Later among the works it cites.
Training and inference with integers in deep neural networks
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi · 2018
Later among the works it cites.
Discrimination-aware channel pruning for deep neural networks
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu · 2018
Later among the works it cites.
Bit-tactical: A software/hardware approach to exploiting value and bit sparsity in neural networks
Alberto Delmas Lascorz, Patrick Judd, Dylan Malone Stuart, Zissis Poulos, Mostafa Mahmoud, Sayeh Sharify, Milos Nikolic, Kevin Siu, and Andreas Moshovos · 2019
Later among the works it cites.
Sparse networks from scratch: Faster training without losing performance
Original
Tim Dettmers and Luke Zettlemoyer · 2019
Later among the works it cites.
Approximated oracle filter pruning for destructive cnn width optimization
Original
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan · 2019
Later among the works it cites.
Rigging the lottery: Making all tickets winners, 2019
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen · 2019
Later among the works it cites.
Accelerating convolutional neural networks via activation map compression
Georgios Georgiadis · 2019
Later among the works it cites.
Dynamic sparse graph for efficient deep learning
Liu Liu, Lei Deng, Xing Hu, Maohua Zhu, Guoqi Li, Yufei Ding, and Yuan Xie · 2019
Later among the works it cites.
Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
Hesham Mostafa and Xin Wang · 2019
Later among the works it cites.
Collaborative channel pruning for deep networks
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang · 2019
Later among the works it cites.
Energy and policy considerations for deep learning in nlp
Original
Emma Strubell, Ananya Ganesh, and Andrew McCallum · 2019
Later among the works it cites.
Once-for-all: Train one network and specialize it for efficient deployment
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han · 2020
Closest in time.
Resprop: Reuse sparsified backpropagation
Negar Goli and Tor Aamodt · 2020
Closest in time.
Soft threshold weight reparameterization for learnable sparsity
Original
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi · 2020
Closest in time.
Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers
Junjie Liu, Zhe XU, Runbin SHI, Ray C. C. Cheung, and Hayden K.H. So · 2020
Closest in time.
NVIDIA A100 Tensor Core GPU Architecture
NVIDIA Corporation · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…