Replacing softmax with ReLU in Vision Transformers

Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories

Li Fei-Fei, Rob Fergus, and Pietro Perona · 2004

Earlier work this paper cites.

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009

Earlier work this paper cites.

Learning multiple layers of features from tiny images, 2009

Alex Krizhevsky, Geoffrey Hinton, et al · 2009

Earlier work this paper cites.

An image is worth 16x16 words: Transformers for image recognition at scale

Original

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2010

Earlier work this paper cites.

Caltech-ucsd birds 200

Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona · 2010

Earlier work this paper cites.

Bag-of-visual-words and spatial extensions for land-use classification

Yi Yang and Shawn Newsam · 2010

Earlier work this paper cites.

Cats and dogs

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar · 2012

Earlier work this paper cites.

Improving deep neural networks for lvcsr using rectified linear units and dropout

George E Dahl, Tara N Sainath, and Geoffrey E Hinton · 2013

Earlier work this paper cites.

3d object representations for fine-grained categorization

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei · 2013

Earlier work this paper cites.

Neural machine translation by jointly learning to align and translate

Original

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014

Earlier work this paper cites.

Describing textures in the wild

Original

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi · 2014

Cited alongside, same era.

Layer normalization

Original

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016

Cited alongside, same era.

Gaussian error linear units (gelus)

Original

Dan Hendrycks and Kevin Gimpel · 2016

Cited alongside, same era.

Collection of textures in colorectal cancer histology

Jakob Nikolas Kather, Frank Gerrit Zöllner, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Cleo-Aron Weis · 2016

Cited alongside, same era.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017

Cited alongside, same era.

Better plain vit baselines for imagenet-1k

Original

Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov · 2022

Later among the works it cites.

Flashattention: Fast and memory-efficient exact attention with io-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022

Later among the works it cites.

Transformer quality in linear time

Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le · 2022

Later among the works it cites.

Sima: Simple softmax-free attention for vision transformers

Original

Soroush Abbasi Koohpayegani and Hamed Pirsiavash · 2022

Later among the works it cites.

Robust training of neural networks using scale invariant architectures

Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar · 2022

Later among the works it cites.