2020

Widening and Squeezing: Towards Accurate and Efficient QNNs

Liu, Chuanjian, Han, Kai, Wang, Yunhe et al.

Understand

Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters.

Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques.
However, we find the representation capability of quantization features is far weaker than full-precision features by experiments.
We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.

Built on

Comparing biases for minimal network construction with back-propagation
Stephen José Hanson and Lorien Pratt · 1989
Earlier work this paper cites.
Second order derivatives for network pruning: optimal brain surgeon
Babak Hassibi and David G Stork · 1993
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Nonlinear dimensionality reduction by locally linear embedding
Sam T Roweis and Lawrence K Saul · 2000
Earlier work this paper cites.
A global geometric framework for nonlinear dimensionality reduction
Joshua B Tenenbaum, Vin De Silva, and John C Langford · 2000
Earlier work this paper cites.
Locality preserving projections
Xiaofei He and Partha Niyogi · 2004
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
The pascal visual object classes challenge 2012 (voc2012) development kit
Mark Everingham and John Winn · 2011
Earlier work this paper cites.
Improving the speed of neural networks on cpus
Vincent Vanhoucke, Andrew Senior, and Mark Z Mao · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Semantic image segmentation with deep convolutional nets and fully connected crfs
Original
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik · 2014
Earlier work this paper cites.
Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1
Kyuyeon Hwang and Wonyong Sung · 2014
Earlier work this paper cites.
Fitnets: Hints for thin deep nets
Original
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio · 2014
Earlier work this paper cites.
Circulant binary embedding
Felix Yu, Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang · 2014
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David · 2015
Earlier work this paper cites.
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally · 2015
Earlier work this paper cites.

Similar

Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Cited alongside, same era.
Speeding-up convolutional neural networks using fine-tuned cp-decomposition
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky · 2015
Cited alongside, same era.
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1
Original
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio · 2016
Cited alongside, same era.
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Song Han, Huizi Mao, and William J Dally · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size
Original
Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer · 2016
Cited alongside, same era.
Compression of deep convolutional neural networks for fast and low power mobile applications
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin · 2016
Cited alongside, same era.

Then

Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, 2016
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou · 2016
Later among the works it cites.
Trained ternary quantization, 2016
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally · 2016
Later among the works it cites.
Mask r-cnn
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick · 2017
Later among the works it cites.
Channel pruning for accelerating very deep neural networks
Yihui He, Xiangyu Zhang, and Jian Sun · 2017
Later among the works it cites.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam · 2017
Later among the works it cites.
Towards accurate binary convolutional neural network
Xiaofan Lin, Cong Zhao, and Wei Pan · 2017
Later among the works it cites.
Learning efficient convolutional networks through network slimming
Zhuang Liu, Jianguo Li, B Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang · 2017
Later among the works it cites.
Bnn+: Improved binary network training
Original
Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia · 2018
Later among the works it cites.
Soft filter pruning for accelerating deep convolutional neural networks
Original
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang · 2018
Later among the works it cites.
Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng · 2018
Later among the works it cites.
Lq-nets: Learned quantization for highly accurate and compact deep neural networks
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua · 2018
Later among the works it cites.
Shufflenet: An extremely efficient convolutional neural network for mobile devices
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun · 2018
Later among the works it cites.
Filter pruning via geometric median for deep convolutional neural networks acceleration
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang · 2019
Later among the works it cites.
Learning instance-wise sparsity for accelerating deep models
Chuanjian Liu, Yunhe Wang, Kai Han, Chunjing Xu, and Chang Xu · 2019
Later among the works it cites.
Searching for accurate binary neural architectures
Mingzhu Shen, Kai Han, Chunjing Xu, and Yunhe Wang · 2019
Later among the works it cites.
Binary ensemble neural network: More bits per network or more networks per bit?
Shilin Zhu, Xin Dong, and Hao Su · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…