Memformer: A Memory-Augmented Transformer for Sequence Modeling

Learning Representations by Back-Propagating Errors , page 696–699. MIT Press, Cambridge, MA, USA

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988 · 1988

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997

Earlier work this paper cites.

Learning to forget: Continual prediction with LSTM

Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. 2000 · 2000

Earlier work this paper cites.

Longformer: The long-document transformer

Original

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 · 2004

Earlier work this paper cites.

Linformer: Self-attention with linear complexity

Original

Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020 · 2006

Earlier work this paper cites.

Rethinking attention with performers

Original

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020 · 2009

Earlier work this paper cites.

MNIST handwritten digit database

Yann LeCun and Corinna Cortes. 2010 · 2010

Earlier work this paper cites.

Empirical evaluation of gated recurrent neural networks on sequence modeling

Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014

Earlier work this paper cites.

Neural turing machines

Original

Alex Graves, Greg Wayne, and Ivo Danihelka. 2014 · 2014

Cited alongside, same era.

Neural machine translation by jointly learning to align and translate

Original

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015

Cited alongside, same era.

Training deep nets with sublinear memory cost

Original

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016 · 2016

Cited alongside, same era.

Hybrid computing using a neural network with dynamic external memory

Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John P. Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. 2016 · 2016

Cited alongside, same era.

Pointer sentinel mixture models

Generating long sequences with sparse transformers

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019 · 2019

Later among the works it cites.

Transformer-xl: Attentive language models beyond a fixed-length context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019 · 2019

Later among the works it cites.

Transformers are rnns: Fast autoregressive transformers with linear attention

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020 · 2020

Closest in time.

Reformer: The efficient transformer

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020 · 2020

Closest in time.

Compressive transformers for long-range sequence modelling

Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020 · 2020

Closest in time.

Big bird: Transformers for longer sequences

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020 · 2020