Built on

Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J · 2002
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al · 2007
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Sequence level training with recurrent neural networks
Original
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Sennrich, R., Haddow, B., and Birch, A · 2015
Earlier work this paper cites.
End-to-end memory networks
Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
An actor-critic algorithm for sequence prediction
Original
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., and Bengio, Y · 2016
Earlier work this paper cites.
From softmax to sparsemax: A sparse model of attention and multi-label classification
Martins, A. and Astudillo, R · 2016
Earlier work this paper cites.

Similar

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Achieving human parity on automatic chinese to english news translation
Original
Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., Li, M., et al · 2018
Cited alongside, same era.
Transformer-xl: Attentive language models beyond a fixed-length context

Then

Augmenting self-attention with persistent memory
Original
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., and Joulin, A · 2019
Later among the works it cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Later among the works it cites.
Transformer feed-forward layers are key-value memories
Original
Geva, M., Schuster, R., Berant, J., and Levy, O · 2020
Later among the works it cites.
Towards making the most of context in neural machine translation
Original
Zheng, Z., Yue, X., Huang, S., Chen, J., and Birch, A · 2020
Later among the works it cites.
Knowledge neurons in pretrained transformers
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

A Study on ReLU and Softmax in Transformer

Shen, Kai, Guo, Junliang, Tan, Xu et al.

Understand

The Transformer architecture consists of self-attention and feed-forward networks (FFNs) which can be viewed as key-value memories according to previous works.

However, FFN and traditional memory utilize different activation functions (i.e., ReLU and Softmax respectively), which makes them not equivalent.
In this paper, we first rebuild the connections between FFN and key-value memory by conducting extensive studies on ReLU and Softmax, and find they are equivalent when adding an additional layer normalization module on Softmax.
In addition, ReLU outperforms Softmax on both FFN and key-value memory when the number of value slots is large.

Built on

Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J · 2002
Earlier work this paper cites.
Moses: Open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al · 2007
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Sequence level training with recurrent neural networks
Original
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Sennrich, R., Haddow, B., and Birch, A · 2015
Earlier work this paper cites.
End-to-end memory networks
Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
An actor-critic algorithm for sequence prediction
Original
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., and Bengio, Y · 2016
Earlier work this paper cites.
From softmax to sparsemax: A sparse model of attention and multi-label classification
Martins, A. and Astudillo, R · 2016
Earlier work this paper cites.

Similar

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Original
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Achieving human parity on automatic chinese to english news translation
Original
Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., Li, M., et al · 2018
Cited alongside, same era.
Transformer-xl: Attentive language models beyond a fixed-length context

Then

Augmenting self-attention with persistent memory
Original
Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., and Joulin, A · 2019
Later among the works it cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Later among the works it cites.
Transformer feed-forward layers are key-value memories
Original
Geva, M., Schuster, R., Berant, J., and Levy, O · 2020
Later among the works it cites.
Towards making the most of context in neural machine translation
Original
Zheng, Z., Yue, X., Huang, S., Chen, J., and Birch, A · 2020
Later among the works it cites.
Knowledge neurons in pretrained transformers
Original