Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

A. Vaswani, N. Shazeer, N. Parmar et al. , “Attention is all you need,” in Proc. of NeurIPS , 2017, pp. 5998–6008

2017

Earlier work this paper cites.

“NVIDIA Tesla V100 GPU Architecture,” NVIDIA, Tech. Rep., 2017

2017

Earlier work this paper cites.

N. P. Jouppi, C. Young, N. Patil et al. , “In-datacenter performance analysis of a tensor processing unit,” in Proc. of ISCA , 2017, pp. 1–12

2017

Earlier work this paper cites.

M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” arXiv preprint arXiv:1805.02867 , 2018

Original

2018

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of NAACL-HLT , 2019, pp. 4171–4189

2019

Earlier work this paper cites.

M. Shoeybi, M. Patwary, R. Puri et al. , “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” arXiv preprint arXiv:1909.08053 , 2019

Original

2019

Earlier work this paper cites.

A. Radford, J. Wu, R. Child et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108 , 2019

Original

2019

Earlier work this paper cites.

Q. Guo, X. Qiu, P. Liu et al. , “Star-Transformer,” in NAACL-HLT , 2019

2019

Cited alongside, same era.

O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8BERT: Quantized 8bit BERT,” arXiv preprint arXiv:1910.06188 , 2019

Original

2019

Cited alongside, same era.

G. Du, C. Tian, Z. Li et al. , “Efficient softmax hardware architecture for deep neural networks,” in Proc. of GLSVLSI , 2019, pp. 75–80

2019

Cited alongside, same era.

R. Venkatesan et al. , “MAGNet: A modular aaccelerator generator for neural networks.” in Proc. of ICCAD , 2019, pp. 1–8

2019

Cited alongside, same era.

T. Wolf et al. , “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv , pp. arXiv–1910, 2019

2019

Cited alongside, same era.

A. Nagarajan, S. Sen, J. R. Stevens, and A. Raghunathan, “Optimizing Transformers with Approximate Computing for Faster, Smaller and more Accurate NLP Models,” arXiv preprint arXiv:2010.03688 , 2020

Original

2020

Later among the works it cites.

G. Prato, E. Charlaix, and M. Rezagholizadeh, “Fully quantized transformer for machine translation,” in Proc. of EMNLP , 2020

2020

Later among the works it cites.

Y. Lin, Y. Li, T. Liu et al. , “Towards fully 8-bit integer inference for the transformer model,” in Proc. of IJCAI , 2020, pp. 3759–3765

2020

Later among the works it cites.

Y. Gao, W. Liu, and F. Lombardi, “Design and implementation of an approximate softmax layer for deep neural networks,” in Proc. of ISCAS . IEEE, 2020, pp. 1–5

2020

Later among the works it cites.

T. J. Ham, S. J. Jung, S. Kim et al. , “Aˆ3: Accelerating attention mechanisms in neural networks with approximation,” in Proc. of HPCA . IEEE, 2020, pp. 328–341