Fetching the paper…

Reading the bibliography…

DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks · Around

2019

DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

Wiedemann, Simon, Kirchoffer, Heiner, Matlage, Stefan et al.

Original
PDF
Post on X

Understand

The field of video compression has developed some of the most sophisticated and efficient compression algorithms known in the literature, enabling very high compressibility for little loss of information.

Whilst some of these techniques are domain specific, many of their underlying principles are universal in that they can be adapted and applied for compressing different types of data.
In this work we present DeepCABAC, a compression algorithm for deep neural networks that is based on one of the state-of-the-art video coding techniques.
Concretely, it applies a Context-based Adaptive Binary Arithmetic Coder (CABAC) to the network's parameters, which was originally designed for the H.264/AVC video coding standard and became the state-of-the-art for lossless compression.

Built on

D. A. Huffman, “A method for the construction of minimum-redundancy codes,” Proceedings of the IRE , vol. 40, no. 9, pp. 1098–1101, Sep. 1952
1952
Earlier work this paper cites.
I. S. Duff, “A survey of sparse matrix research,” Proceedings of the IEEE , vol. 65, no. 4, pp. 500–535, 1977
1977
Earlier work this paper cites.
J. Teuhola, “A compression method for clustered bit-vectors,” Information Processing Letters , vol. 7, no. 6, pp. 308–311, 1978
1978
Earlier work this paper cites.
Y. L. Cun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in Neural Information Processing Systems (NIPS) , 1990, pp. 598–605
1990
Earlier work this paper cites.
B. Hassibi, D. G. Stork, and G. J. Wolff, “Optimal brain surgeon and general network pruning,” in IEEE International Conference on Neural Networks , 1993, pp. 293–299
1993
Earlier work this paper cites.
C. E. Shannon, “A mathematical theory of communication,” SIGMOBILE Mobile Computing and Communications Review , vol. 5, no. 1, pp. 3–55, 2001
2001
Earlier work this paper cites.
D. Marpe, H. Schwarz, and T. Wiegand, “Context-based adaptive binary arithmetic coding in the h.264/avc video compression standard,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 13, no. 7, pp. 620–636, July 2003
2003
Earlier work this paper cites.
D. Marpe and T. Wiegand, “A highly efficient multiplication-free binary arithmetic coder and its application in video coding,” in Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429) , vol. 2, Sep. 2003, pp. 263–266
2003
Earlier work this paper cites.
P. Grünwald and J. Rissanen, The Minimum Description Length Principle , ser. Adaptive computation and machine learning. MIT Press, 2007
2007
Earlier work this paper cites.
T. Wiegand and H. Schwarz, “Source coding: Part 1 of fundamentals of source and video coding,” Found. Trends Signal Process. , vol. 4, no. 1–2, pp. 1–222, 2011
2011
Earlier work this paper cites.
M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) , Feb 2014, pp. 10–14
2014
Earlier work this paper cites.
V. Sze, M. Budagavi, and G. J. Sullivan, High Efficiency Video Coding: Algorithms and Architectures . Springer, 2014
2014
Earlier work this paper cites.
S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” arXiv:1410.0759 , 2014
Original
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
J. Martens, “New perspectives on the natural gradient method,” arXiv:1412.1193 , 2014
Original
2014
Earlier work this paper cites.
Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” Nature , vol. 521, pp. 436–444, 2015
2015
Earlier work this paper cites.
J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks , vol. 61, pp. 85–117, 2015
2015
Earlier work this paper cites.
S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in Advances in Neural Information Processing Systems (NIPS) , 2015, pp. 1135–1143
2015
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,” arXiv:1510.00149 , 2015
Original
2015
Earlier work this paper cites.
S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” arXiv:1502.02551 , 2015
Original
2015
Earlier work this paper cites.

Similar

D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,” in Advances in Neural Information Processing Systems (NIPS) , 2015, pp. 2575–2583
2015
Cited alongside, same era.
H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” arXiv:1602.05629 , 2016
Original
2016
Cited alongside, same era.
Y. Guo, A. Yao, and Y. Chen, “Dynamic network surgery for efficient dnns,” arXiv:1608.04493 , 2016
Original
2016
Cited alongside, same era.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” arXiv:1603.05279 , 2016
Original
2016
Cited alongside, same era.
M. Courbariaux and Y. Bengio, “Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 , 2016

Then

A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental network quantization: Towards lossless cnns with low-precision weights,” arXiv:1702.03044 , 2017
Original
2017
Later among the works it cites.
M. Federici, K. Ullrich, and M. Welling, “Improved Bayesian Compression,” arXiv:1711.06494 , 2017
Original
2017
Later among the works it cites.
X. Xiaowei, D. Yukun, H. S. Xiaobo, N. Michael, C. Jason, H. Yu, and S. Yiyu, “Scaling for edge inference of deep neural networks,” Nature Electronics , vol. 1, pp. 216–222, Apr 2018
2018
Later among the works it cites.
Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “Model compression and acceleration for deep neural networks: The principles, progress, and challenges,” IEEE Signal Processing Magazine , vol. 35, no. 1, pp. 126–136, Jan 2018
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2016

Cited alongside, same era.

C. Zhu, S. Han, H. Mao, and W. J. Dally, “Trained ternary quantization,” arXiv:1612.01064 , 2016

2016

Cited alongside, same era.

F. Li, B. Zhang, and B. Liu, “Ternary weight networks,” arXiv:1605.04711 , 2016

2016

Cited alongside, same era.

M. Tu, V. Berisha, Y. Cao, and J. S. Seo, “Reducing the model order of deep neural networks using information theory,” in 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI) , 2016, pp. 93–98

2016

Cited alongside, same era.

Y. Choi, M. El-Khamy, and J. Lee, “Towards the limit of network quantization,” arXiv:1612.01543 , 2016

2016

Cited alongside, same era.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861 , 2017

2017

Cited alongside, same era.

K. Ota, M. S. Dao, V. Mezaris, and F. G. B. D. Natale, “Deep learning for mobile multimedia: A survey,” ACM Trans. Multimedia Comput. Commun. Appl. , vol. 13, no. 3s, pp. 34:1–34:22, Jun. 2017

2017

Cited alongside, same era.

Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and acceleration for deep neural networks,” arXiv:1710.09282 , 2017

Original

2017

Cited alongside, same era.

E. Tartaglione, S. Lepsø y, A. Fiandrotti, and G. Francini, “Learning sparse neural networks via sensitivity-driven regularization,” in Advances in Neural Information Processing Systems (NIPS) , 2018, pp. 3878–3888

2018

Later among the works it cites.

J. Achterhold, J. M. Köhler, A. Schmeink, and T. Genewein, “Variational network quantization,” in International Conference on Representation Learning (ICLR) , 2018

2018

Later among the works it cites.

Y. Choi, M. El-Khamy, and J. Lee, “Universal deep neural network compression,” arXiv:1802.02271 , 2018

2018

Later among the works it cites.

R. Banner, I. Hubara, E. Hoffer, and D. Soudry, “Scalable methods for 8-bit training of neural networks,” in Advances in Neural Information Processing Systems (NIPS) , 2018, pp. 5145–5153

2018

Later among the works it cites.

N. Wang, J. Choi, D. Brand, C. Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point numbers,” arXiv:1812.08011 , 2018

2018

Later among the works it cites.

C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” IEEE Communications Surveys Tutorials , pp. 1–1, 2019

2019

Closest in time.

F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Sparse binary compression: Towards distributed deep learning with minimal communication,” in IEEE International Joint Conference on Neural Networks (IJCNN) , 2019

2019

Closest in time.

F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data,” arXiv:1903.02891 , 2019

2019

Closest in time.

S. Wiedemann, K.-R. Müller, and W. Samek, “Compact and computationally efficient representation of deep neural networks,” IEEE Transactions on Neural Networks and Learning Systems , 2019

2019

Closest in time.

MPEG Requirements, “Updated call for proposals on neural network compression,” Moving Picture Experts Group (MPEG), Marrakech, MA, CfP, Jan. 2019

2019

Closest in time.

“QNNPACK open source library for optimized mobile deep learning,” https://github.com/pytorch/QNNPACK

2019

Closest in time.

“TensorFlow Lite,” https://www.tensorflow.org/lite

2019

Closest in time.

S. Wiedemann, A. Marbán, K.-R. Müller, and W. Samek, “Entropy-constrained training of deep neural networks,” in IEEE International Joint Conference on Neural Networks (IJCNN) , 2019

2019

Closest in time.

J. Seward, “bzip2,” http://www.bzip.org

2019

Closest in time.

S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, “Unmasking clever hans predictors and assessing what machines really learn,” Nature Communications , vol. 10, p. 1096, 2019

2019

Closest in time.