2017

Soft Weight-Sharing for Neural Network Compression

Ullrich, Karen, Meeds, Edward, Welling, Max

Understand

The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices.

This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression.
Recent work by Han et al.
(2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates.

Built on

A stochastic approximation method
Herbert Robbins and Sutton Monro · 1951
Earlier work this paper cites.
Modeling by shortest data description
Jorma Rissanen · 1978
Earlier work this paper cites.
Stochastic complexity and modeling
Jorma Rissanen · 1986
Earlier work this paper cites.
Arithmetic coding for data compression
Ian H Witten, Radford M Neal, and John G Cleary · 1987
Earlier work this paper cites.
Optimal brain damage
Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel · 1989
Earlier work this paper cites.
Classification by minimum-message-length inference
Chris S Wallace · 1990
Earlier work this paper cites.
Simplifying neural networks by soft weight-sharing
Steven J Nowlan and Geoffrey E Hinton · 1992
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
Geoffrey E Hinton and Drew Van Camp · 1993
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Variational learning and bits-back coding: an information-theoretic view to bayesian learning
Antti Honkela and Harri Valpola · 2004
Earlier work this paper cites.
Pattern recognition
Christopher M Bishop · 2006
Earlier work this paper cites.
Practical variational inference for neural networks
Alex Graves · 2011
Earlier work this paper cites.
Multiresolution mixture modeling using merging of mixture components
Prem Raj Adhikari and Jaakko Hollmén · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams · 2012
Earlier work this paper cites.

Similar

Predicting parameters in deep learning
Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al · 2013
Cited alongside, same era.
Training deep neural networks with low precision multiplications
Original
Matthieu Courbariaux, Jean-Pierre David, and Yoshua Bengio · 2014
Cited alongside, same era.
Exploiting linear structure within convolutional networks for efficient evaluation
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus · 2014
Cited alongside, same era.
Compressing deep convolutional networks using vector quantization
Original
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev · 2014
Cited alongside, same era.
Speeding up convolutional neural networks with low rank expansions
Original
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Compressing convolutional neural networks
Original
Wenlin Chen, James T Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen · 2015
Cited alongside, same era.
Binaryconnect: Training deep neural networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David · 2015
Cited alongside, same era.
Deep learning with limited numerical precision
Original
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Cited alongside, same era.
Training cnns with low-rank filters for efficient image classification
Original
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi · 2015
Cited alongside, same era.

Then

Sparse convolutional neural networks
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky · 2015
Later among the works it cites.
Convolutional neural networks with low-rank regularization
Original
Cheng Tai, Tong Xiao, Xiaogang Wang, et al · 2015
Later among the works it cites.
Quantized convolutional neural networks for mobile devices
Original
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng · 2015
Later among the works it cites.
Binarynet: Training deep neural networks with weights and activations constrained to + 1 +1 or − 1 -1
Original
Matthieu Courbariaux and Yoshua Bengio · 2016
Later among the works it cites.
Binarized neural networks: Training neural networks with weights and activations constrained to + 1 +1 or − 1 -1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio · 2016
Later among the works it cites.
Dynamic network surgery for efficient dnns
Yiwen Guo, Anbang Yao, and Yurong Chen · 2016
Later among the works it cites.
Identity mappings in deep residual networks
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 1mb model size
Original
Forrest N Iandola, Matthew W Moskewicz, Khalid Ashraf, Song Han, William J Dally, and Kurt Keutzer · 2016
Later among the works it cites.
Deep neural networks are robust to weight binarization and other non-linear distortions
Original
Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve K Esser, and Dharmendra Modha · 2016
Later among the works it cites.
Accelerating deep convolutional networks using low-precision and sparsity
Original
Ganesh Venkatesh, Eriko Nurvitadhi, and Debbie Marr · 2016
Later among the works it cites.
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Later among the works it cites.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Later among the works it cites.
Variational dropout sparsifies deep neural networks
Original
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…