Fetching the paper…

Reading the bibliography…

2020

Faster Transformer Decoding: N-gram Masked Self-Attention

Chelba, Ciprian, Chen, Mia, Bapna, Ankur et al.

Original
PDF
Post on X

Understand

Motivated by the fact that most of the information relevant to the prediction of target tokens is drawn from the source sentence $S=s_1, \ldots, s_S$, we propose truncating the target-side window used for computing self-attention by making an $N$-gram assumption.

Experiments on WMT EnDe and EnFr data sets show that the $N$-gram masked self-attention model loses very little in BLEU score for $N$ values in the range $4, \ldots, 8$, depending on the task.

Faster Transformer Decoding: N-gram Masked Self-Attention · Around

Built on

Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Original
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, and Yuan Cao et al. 2019 · 1902
Earlier work this paper cites.
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Generating Wikipedia by Summarizing Long Sequences
Original
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018 · 2018
Earlier work this paper cites.

Similar

Image Transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018 · 2018
Cited alongside, same era.
A Time-restricted Self Attention Layer for ASR
Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, and Sanjeev Khudanpur. 2018 · 2018
Cited alongside, same era.
Alibaba Submission for WMT18 Quality Estimation Task
Jiayi Wang, Kai Fan, Bo Li, Fengming Zhou, Boxing Chen, Yangbin Shi, and Luo Si. 2018 · 2018
Cited alongside, same era.

Then

Accelerating Neural Transformer via an Average Attention Network
Original
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018 · 2018
Later among the works it cites.
Fast Transformer Decoding: One Write-Head is All You Need
Noam Shazeer. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…