GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

Prefix sums and their applications

Guy Blelloch · 1990

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber · 1997

Earlier work this paper cites.

Jointly learning to align and translate with transformer models, 2019

Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik · 2019

Earlier work this paper cites.

Long range arena: A benchmark for efficient transformers, 2020

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler · 2020

Earlier work this paper cites.

Dissecting neural odes, 2021

Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama · 2021

Earlier work this paper cites.

Efficiently modeling long sequences with structured state spaces, 2022

Albert Gu, Karan Goel, and Christopher Ré · 2022

Cited alongside, same era.

In-context learning and induction heads, 2022

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah · 2022

Cited alongside, same era.

A length-extrapolatable transformer, 2022

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei · 2022

Cited alongside, same era.

Hungry hungry hippos: Towards language modeling with state space models, 2023

Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré · 2023

Cited alongside, same era.

Encoding recurrence into transformers

Feiqing Huang, Kexin Lu, Yuxi CAI, Zhen Qin, Yanwen Fang, Guangjian Tian, and Guodong Li · 2023

Cited alongside, same era.

Simplified State Space Layers for Sequence Modeling [source code]

Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman

Cited in the paper.

Simplified state space layers for sequence modeling, 2023b

Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman

Cited in the paper.

Mega: Moving average equipped gated attention, 2023

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer · 2023

Closest in time.

Resurrecting recurrent neural networks for long sequences, 2023

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De · 2023

Closest in time.

Hyena hierarchy: Towards larger convolutional language models, 2023

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré · 2023

Closest in time.

Retentive network: A successor to transformer for large language models, 2023

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei · 2023

Closest in time.

Attention is all you need, 2023

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2023

Closest in time.