On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages

Randomized Positional Encodings Boost Length Generalization of Transformers

Ruoss, A.; Delétang, G.; Genewein, T.; Grau-Moya, J.; Csordás, R.; Bennani, M.; Legg, S.; and Veness, J. 2023 · 1903

Earlier work this paper cites.

Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

Original

Suzgun, M.; Gehrmann, S.; Belinkov, Y.; and Shieber, S. M. 2019 · 1911

Earlier work this paper cites.

Neural Nets and the Brain Model Problem

Minsky, M. 1954 · 1954

Earlier work this paper cites.

Algebraic Theory of Machines. I. Prime Decomposition Theorem for Finite Semigroups and Machines

Krohn, K.; and Rhodes, J. 1965 · 1965

Earlier work this paper cites.

Computation: finite and infinite machines

Minsky, M. L. 1967 · 1967

Earlier work this paper cites.

Threshold matrices and the state assignment problem for neural nets

Dewdney, A. K. 1977 · 1977

Earlier work this paper cites.

Prefix Sums and Their Applications

Blelloch, G. E. 1990 · 1990

Earlier work this paper cites.

Finding Structure in Time

Elman, J. L. 1990 · 1990

Earlier work this paper cites.

Finite automata, formal logic, and circuit complexity

Straubing, H. 1994 · 1994

Earlier work this paper cites.

Optimal simulation of automata by neural nets

Indyk, P. 1995 · 1995

Earlier work this paper cites.

On the Computational Power of Neural Nets

Siegelmann, H.; and Sontag, E. 1995 · 1995

Earlier work this paper cites.

Long Short-Term Memory

Hochreiter, S.; and Schmidhuber, J. 1997 · 1997

Earlier work this paper cites.

Visual group theory

Carter, N. C. 2009 · 2009

Earlier work this paper cites.

Inferring algorithmic patterns with stack-augmented recurrent nets

Joulin, A.; and Mikolov, T. 2015 · 2015

Earlier work this paper cites.

Layer Normalization

Original

Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016 · 2016

Earlier work this paper cites.

Attention Is All You Need

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017 · 2017

Cited alongside, same era.

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

Martin, E.; and Cundy, C. 2018 · 2018

Cited alongside, same era.

On the Ability and Limitations of Transformers to Recognize Formal Languages

Bhattamishra, S.; Ahuja, K.; and Goyal, N. 2020 · 2020

Cited alongside, same era.

HiPPO: Recurrent Memory with Optimal Polynomial Projections

Gu, A.; Dao, T.; Ermon, S.; Rudra, A.; and Re, C. 2020 · 2020

Cited alongside, same era.

Theoretical Limitations of Self-Attention in Neural Sequence Models

Hahn, M. 2020 · 2020

Cited alongside, same era.

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Gu, A.; Johnson, I.; Goel, K.; Saab, K.; Dao, T.; Rudra, A.; and Ré, C. 2021 · 2021

Hyena Hierarchy: Towards Larger Convolutional Language Models

Original

Poli, M.; Massaroli, S.; Nguyen, E.; Fu, D. Y.; Dao, T.; Baccus, S.; Bengio, Y.; Ermon, S.; and Ré, C. 2023 · 2023

Later among the works it cites.

Simplified State Space Layers for Sequence Modeling

Smith, J. T. H.; Warrington, A.; and Linderman, S. W. 2023 · 2023

Later among the works it cites.

Efficiently Representing Finite-state Automata With Recurrent Neural Networks

Original

Svete, A.; and Cotterell, R. 2023 · 2023

Later among the works it cites.

Theoretical Foundations of Deep Selective State-Space Models

Original

Cirone, N. M.; Orvieto, A.; Walker, B.; Salvi, C.; and Lyons, T. 2024 · 2024

Closest in time.

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Dao, T. 2024 · 2024