Built on

Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
A taxonomy of human translation styles
Michael Carl, Barbara Dragsted, and Arnt Lykke Jakobsen. 2011 · 2011
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Neural turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014 · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014
Earlier work this paper cites.
Listen, attend and spell
Original
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2015 · 2015
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Original
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015 · 2015
Earlier work this paper cites.

Similar

Effective approaches to attention-based neural machine translation
Original
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
A neural attention model for abstractive sentence summarization
Original
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015 · 2015
Cited alongside, same era.
Weakly supervised memory networks
Original
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015 · 2015
Cited alongside, same era.
A neural conversational model
Original
Oriol Vinyals and Quoc V. Le. 2015 · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention

Then

A convolutional encoder model for neural machine translation
Original
Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2016 · 2016
Later among the works it cites.
Neural machine translation in linear time
Original
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016 · 2016
Later among the works it cites.
Sequence-to-sequence rnns for text summarization
Original
Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016 · 2016
Later among the works it cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Later among the works it cites.
Sequence-to-sequence learning as beam-search optimization
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2017

Efficient Attention using a Fixed-Size Memory Representation

Britz, Denny, Guan, Melody Y., Luong, Minh-Thang

Understand

The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step.

In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient.
Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory.
We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences.

Built on

Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
A taxonomy of human translation styles
Michael Carl, Barbara Dragsted, and Arnt Lykke Jakobsen. 2011 · 2011
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Neural turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014 · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014 · 2014
Earlier work this paper cites.
Listen, attend and spell
Original
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2015 · 2015
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Original
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015 · 2015
Earlier work this paper cites.

Similar

Effective approaches to attention-based neural machine translation
Original
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015 · 2015
Cited alongside, same era.
A neural attention model for abstractive sentence summarization
Original
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015 · 2015
Cited alongside, same era.
Weakly supervised memory networks
Original
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015 · 2015
Cited alongside, same era.
A neural conversational model
Original
Oriol Vinyals and Quoc V. Le. 2015 · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention

Then

A convolutional encoder model for neural machine translation
Original
Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2016 · 2016
Later among the works it cites.
Neural machine translation in linear time
Original
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aäron van den Oord, Alex Graves, and Koray Kavukcuoglu. 2016 · 2016
Later among the works it cites.
Sequence-to-sequence rnns for text summarization
Original
Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016 · 2016
Later among the works it cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Later among the works it cites.
Sequence-to-sequence learning as beam-search optimization
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015 · 2015

Cited alongside, same era.

Morphological inflection generation with hard monotonic attention

Original

Roee Aharoni and Yoav Goldberg. 2016 · 2016

Cited alongside, same era.

Towards better decoding and language model integration in sequence to sequence models

Original

Jan Chorowski and Navdeep Jaitly. 2016 · 2016

Cited alongside, same era.

A cheap linear attention mechanism with fast lookups and fixed-size representations

Original

Alexandre de Brébisson and Pascal Vincent. 2016 · 2016

Cited alongside, same era.

Sam Wiseman and Alexander M. Rush. 2016 · 2016

Later among the works it cites.

Google’s neural machine translation system: Bridging the gap between human and machine translation

Original

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016 · 2016

Later among the works it cites.

Massive Exploration of Neural Machine Translation Architectures

Original

Denny Britz, Anna Goldie, Thang Luong, and Quoc Le. 2017 · 2017

Closest in time.

Online and linear-time attention by enforcing monotonic alignments

Original

Colin Raffel, Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017 · 2017

Closest in time.