Fetching the paper…

Reading the bibliography…

2021

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

Graham, Ben, El-Nouby, Alaaeldin, Touvron, Hugo et al.

Original
PDF
Post on X

Understand

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime.

Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware.
We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions.
We also introduce the attention bias, a new way to integrate positional information in vision transformers.

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference · Around

Built on

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation , vol. 1, no. 4, pp. 541–551, 1989
1989
Earlier work this paper cites.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Conference on Computer Vision and Pattern Recognition , 2009
2009
Earlier work this paper cites.
Vinod Nair and Geoffrey E Hinton, “Rectified linear units improve restricted boltzmann machines,” in International Conference on Machine Learning , 2010
2010
Earlier work this paper cites.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems , 2012
2012
Earlier work this paper cites.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, “Imagenet large scale visual recognition challenge,” International journal of Computer Vision , 2015
2015
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, “Going deeper with convolutions,” in Conference on Computer Vision and Pattern Recognition , 2015
2015
Earlier work this paper cites.
Sergey Ioffe and Christian Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International Conference on Machine Learning , 2015
2015
Earlier work this paper cites.
Song Han, Huizi Mao, and William J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2016
Original
2016
Earlier work this paper cites.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David, “Binaryconnect: Training deep neural networks with binary weights during propagations,” arXiv preprint arXiv:1511.00363 , 2016
Original
2016
Earlier work this paper cites.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Conference on Computer Vision and Pattern Recognition , 2016
2016
Earlier work this paper cites.
Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke, “Inception-v4, inception-resnet and the impact of residual connections on learning,” arXiv preprint arXiv:1602.07261 , 2016
Original
2016
Earlier work this paper cites.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger, “Deep networks with stochastic depth,” in European Conference on Computer Vision , 2016
2016
Earlier work this paper cites.
Sergey Zagoruyko and Nikos Komodakis, “Wide residual networks,” arXiv preprint arXiv:1605.07146 , 2016
Original
2016
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He, “Aggregated residual transformations for deep neural networks,” Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter, “A downsampled variant of imagenet as an alternative to the cifar datasets,” arXiv preprint arXiv:1707.08819 , 2017
Original
2017
Earlier work this paper cites.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin, “Convolutional sequence to sequence learning,” arXiv preprint arXiv:1705.03122 , 2017
Original
2017
Earlier work this paper cites.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang, “Residual attention network for image classification,” in Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.
Jie Hu, Li Shen, and Gang Sun, “Squeeze-and-excitation networks,” arXiv preprint arXiv:1709.01507 , 2017
Original
2017
Earlier work this paper cites.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger, “Densely connected convolutional networks,” in Conference on Computer Vision and Pattern Recognition , 2017
2017
Earlier work this paper cites.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017
Original
2017
Earlier work this paper cites.

Similar

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou, “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160 , 2018
Original
2018
Cited alongside, same era.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Cited alongside, same era.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, “Improving language understanding with unsupervised learning,” 2018
2018
Cited alongside, same era.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran, “Image transformer,” in International Conference on Machine Learning . PMLR, 2018, pp. 4055–4064
2018
Cited alongside, same era.

Then

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei et al. , “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in European Conference on Computer Vision , 2020
2020
Later among the works it cites.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry, “Augment your batch: Improving generalization through instance repetition,” in Conference on Computer Vision and Pattern Recognition , 2020
2020
Later among the works it cites.
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk, “Toward transformer-based object detection,” arXiv preprint arXiv:2012.09958 , 2020
Original
2020
Later among the works it cites.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr et al. , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” arXiv preprint arXiv:2012.15840 , 2020
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani, “Self-attention with relative position representations,” arXiv preprint arXiv:1803.02155 , 2018

2018

Cited alongside, same era.

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon, “Cbam: Convolutional block attention module,” in European Conference on Computer Vision , 2018, pp. 3–19

2018

Cited alongside, same era.

X. Wang, Ross B. Girshick, A. Gupta, and Kaiming He, “Non-local neural networks,” Conference on Computer Vision and Pattern Recognition , 2018

2018

Cited alongside, same era.

Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han, “Haq: Hardware-aware automated quantization with mixed precision,” arXiv preprint arXiv:1811.08886 , 2019

2019

Cited alongside, same era.

Mingxing Tan and Quoc V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” arXiv preprint arXiv:1905.11946 , 2019

2019

Cited alongside, same era.

A. Howard, Mark Sandler, G. Chu, Liang-Chieh Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Quoc V. Le, and H. Adam, “Searching for MobileNetV3,” in International Conference on Computer Vision , 2019

2019

Cited alongside, same era.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509 , 2019

2019

Cited alongside, same era.

Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze, “Multigrain: a unified image embedding for classes and instances,” arXiv preprint arXiv:1902.05509 , 2019

2019

Cited alongside, same era.

2020

Later among the works it cites.

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu, “Dynamic convolution: Attention over convolution kernels,” in Conference on Computer Vision and Pattern Recognition , 2020

2020

Later among the works it cites.

Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun, “Exploring self-attention for image recognition,” in Conference on Computer Vision and Pattern Recognition , 2020

2020

Later among the works it cites.

Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola, “Resnest: Split-attention networks,” arXiv preprint arXiv:2004.08955 , 2020

Original

2020

Later among the works it cites.

Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda, “Visual transformers: Token-based image representation and processing for computer vision,” arXiv preprint arXiv:2006.03677 , 2020

2020

Later among the works it cites.

Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli, Xuhui Jia, and Ching-Hui Chen, “Global self-attention networks for image recognition,” arXiv preprint arXiv:2010.03019 , 2020

2020

Later among the works it cites.

Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord, “Are we done with imagenet?” arXiv preprint arXiv:2006.07159 , 2020

2020

Later among the works it cites.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou, “Training data-efficient image transformers & distillation through attention,” arXiv preprint arXiv:2012.12877 , 2021

2021

Closest in time.

Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou, “Training vision transformers for image retrieval,” arXiv preprint arXiv:2102.05644 , 2021

2021

Closest in time.

Irwan Bello, “Lambdanetworks: Modeling long-range interactions without attention,” arXiv preprint arXiv:2102.08602 , 2021

2021

Closest in time.

A. Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, P. Abbeel, and Ashish Vaswani, “Bottleneck transformers for visual recognition,” arXiv preprint arXiv:2101.11605 , 2021

2021

Closest in time.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” arXiv preprint arXiv:2102.12122 , 2021

2021

Closest in time.

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” arXiv preprint arXiv:2101.11986 , 2021

2021

Closest in time.

Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia, “Do we really need explicit position encodings for vision transformers?” arXiv preprint arXiv:2102.10882 , 2021

2021

Closest in time.

Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh, “Rethinking spatial dimensions of vision transformers,” 2021

2021

Closest in time.

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, “Cvt: Introducing convolutions to vision transformers,” 2021

2021

Closest in time.

A. Brock, Soham De, S. L. Smith, and K. Simonyan, “High-performance large-scale image recognition without normalization,” arXiv preprint arXiv:2102.06171 , 2021

2021

Closest in time.

Mingxing Tan and Quoc V. Le, “Efficientnetv2: Smaller models and faster training,” 2021

2021

Closest in time.