SP-ViT: Learning 2D Spatial Priors for Vision Transformers

Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems , Vol. 33. 1877–1901

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Attention is All you Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , Vol. 30. 5998–6008

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , Vol. 2. 464–468

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . 4171–4186

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina N. Toutanova. 2018 · 2019

Earlier work this paper cites.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 2019

Earlier work this paper cites.

PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems , Vol. 32. 8026–8037

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019 · 2019

Earlier work this paper cites.

Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning . PMLR, 6105–6114

Mingxing Tan and Quoc Le. 2019 · 2019

Earlier work this paper cites.

XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems , Vol. 32. 5753–5763

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019 · 2019

Earlier work this paper cites.

Transformer Interpretability Beyond Attention Visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 782–791

Hila Chefer, Shir Gur, and Lior Wolf. 2021 · 2021

Earlier work this paper cites.

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 357–366

Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. 2021 · 2021

Earlier work this paper cites.

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Original

Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. 2021 · 2021

Cited alongside, same era.

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. In ICML 2021: 38th International Conference on Machine Learning

Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. 2021 · 2021

Cited alongside, same era.

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Original

Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. 2021 · 2021

Cited alongside, same era.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR 2021: The Ninth International Conference on Learning Representations

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021 · 2021

Cited alongside, same era.

Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 10012–10022

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021 · 2021

Later among the works it cites.

Training data-efficient image transformers & distillation through attention. In ICML 2021: 38th International Conference on Machine Learning

Hugo Touvron, Matthieu Cord, Douze Matthijs, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. 2021a · 2021

Later among the works it cites.

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Original

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021 · 2021

Later among the works it cites.

CvT: Introducing Convolutions to Vision Transformers

Original

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. 2021 · 2021

Later among the works it cites.