2021

Vision Transformer Pruning

Zhu, Mingjian, Tang, Yehui, Han, Kai

Understand

Vision transformer has achieved competitive performance on a variety of computer vision applications.

However, their storage, run-time memory, and computational demands are hindering the deployment to mobile devices.
Here we present a vision transformer pruning approach, which identifies the impacts of dimensions in each layer of transformer and then executes pruning accordingly.
By encouraging dimension-wise sparsity in the transformer, important dimensions automatically emerge.

Built on

Optimal brain damage. In Advances in neural information processing systems . 598–605
Yann LeCun, John S Denker, and Sara A Solla. 1990 · 1990
Earlier work this paper cites.
Second order derivatives for network pruning: Optimal brain surgeon
Babak Hassibi and David G Stork. 1993 · 1993
Earlier work this paper cites.
Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems . 1269–1277
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. 2014 · 2014
Earlier work this paper cites.
Deep learning with limited numerical precision. In International Conference on Machine Learning . 1737–1746
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015 · 2015
Earlier work this paper cites.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015
Earlier work this paper cites.
Fitnets: Hints for thin deep nets. In International Conference on Learning Representations
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Earlier work this paper cites.
Convolutional neural networks with low-rank regularization
Original
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al · 2015
Earlier work this paper cites.
Face model compression by distilling knowledge from neurons. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 30
Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang. 2016 · 2016
Earlier work this paper cites.
Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision . Springer, 525–542
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016 · 2016
Earlier work this paper cites.
Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5918–5926
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. 2017 · 2017
Earlier work this paper cites.
Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision . 2736–2744
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
On compressing deep models by low rank and sparse decomposition. In IEEE Conference on Computer Vision and Pattern Recognition . 7370–7379
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017 · 2017
Earlier work this paper cites.
mixup: Beyond empirical risk minimization
Original
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017 · 2017
Earlier work this paper cites.
Holistic cnn compression via low-rank decomposition with knowledge transfer
Shaohui Lin, Rongrong Ji, Chao Chen, Dacheng Tao, and Jiebo Luo. 2018 · 2018
Earlier work this paper cites.

Similar

Fixing weight decay regularization in adam
Ilya Loshchilov and Frank Hutter. 2018 · 2018
Cited alongside, same era.
Star-transformer
Original
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019 · 2019
Cited alongside, same era.
Tinybert: Distilling bert for natural language understanding
Original
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019 · 2019
Cited alongside, same era.
Albert: A lite bert for self-supervised learning of language representations
Original
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019 · 2019
Cited alongside, same era.
Learning low-rank approximation for cnns
Original
Dongsoo Lee, Se Jung Kwon, Byeongwook Kim, and Gu-Yeon Wei. 2019 · 2019
Cited alongside, same era.
Are sixteen heads really better than one?
Original
Paul Michel, Omer Levy, and Graham Neubig. 2019 · 2019
Cited alongside, same era.
Patient knowledge distillation for bert model compression
Original
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019 · 2019
Cited alongside, same era.
Contrastive representation distillation
Original
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019 · 2019
Cited alongside, same era.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 6023–6032
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019 · 2019
Cited alongside, same era.
Q8bert: Quantized 8bit bert
Original
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019 · 2019
Cited alongside, same era.
End-to-End Object Detection with Transformers
Original
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020a · 2020
Cited alongside, same era.
Generative pretraining from pixels. In International Conference on Machine Learning . PMLR, 1691–1703
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020 · 2020
Cited alongside, same era.

Then

Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 8815–8821
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020 · 2020
Later among the works it cites.
SCOP: Scientific Control for Reliable Neural Network Pruning
Original
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. 2020a · 2020
Later among the works it cites.
Training data-efficient image transformers & distillation through attention
Original
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2020 · 2020
Later among the works it cites.
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Original
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen. 2020b · 2020
Later among the works it cites.
End-to-End Video Instance Segmentation with Transformers
Original
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2020a · 2020
Later among the works it cites.
Searching for low-bit weights in quantized neural networks. In NeurIPS
Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. 2020 · 2020
Later among the works it cites.
Big bird: Transformers for longer sequences
Original
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al · 2020
Later among the works it cites.
An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Models. In CCF International Conference on Natural Language Processing and Chinese Computing . Springer, 359–371
Zihan Zhao, Yuncong Liu, Lu Chen, Qi Liu, Rao Ma, and Kai Yu. 2020 · 2020
Later among the works it cites.
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Original
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020 · 2020
Later among the works it cites.
Pre-trained image processing transformer. In CVPR
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. 2021 · 2021
Closest in time.
An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al · 2021
Closest in time.
Transformer in transformer
Original
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021 · 2021
Closest in time.
ISTR: End-to-End Instance Segmentation with Transformers
Original
Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. 2021 · 2021
Closest in time.
Swin transformer: Hierarchical vision transformer using shifted windows
Original
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021 · 2021
Closest in time.
Pyramid vision transformer: A versatile backbone for dense prediction without convolutions
Original
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021 · 2021
Closest in time.
Learning structured sparsity in deep neural networks. In NeurIPS . 2074–2082
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016 · 2082
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…