Fetching the paper…

Reading the bibliography…

2019

Fast Transformer Decoding: One Write-Head is All You Need

Shazeer, Noam

Original
PDF
Post on X

Understand

Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences.

While training these layers is generally fast and simple, due to parallelizability across the length of the sequence, incremental inference (where such paralleization is impossible) is often slow, due to the memory-bandwidth cost of repeatedly loading the large "keys" and "values" tensors.
We propose a variant called multi-query attention, where the keys and values are shared across all of the different attention "heads", greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding.
We verify experimentally that the resulting models can indeed be much faster to decode, and incur only minor quality degradation from the baseline.

Fast Transformer Decoding: One Write-Head is All You Need · Around

Built on

One billion word benchmark for measuring progress in statistical language modeling
Original
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn · 2013
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate, 2014
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.

Similar

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Generating wikipedia by summarizing long sequences
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer · 2018
Cited alongside, same era.

Then

A time-restricted self-attention layer for ASR
Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, and Sanjeev Khudanpur · 2018
Later among the works it cites.
Accelerating neural transformer via an average attention network, 2018
Biao Zhang, Deyi Xiong, and Jinsong Su · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…