State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

Learning long-term dependencies with gradient descent is difficult

Y. Bengio, P. Simard, and P. Frasconi · 1941

Earlier work this paper cites.

On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition , volume 28, pages 55–59

A. N. Kolmogorov · 1963

Earlier work this paper cites.

ON THE STRUCTURE OF REPRESENTATIONS OF CONTINUOUS FUNCTIONS OF SEVERAL VARIABLES AS FINITE SUMS OF CONTINUOUS FUNCTIONS OF ONE VARIABLE

David A Sprecher · 1965

Earlier work this paper cites.

Analytical Foundations of Volterra Series

Stephen Boyd, L. O. Chua, and C. A. Desoer · 1984

Earlier work this paper cites.

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1986

Earlier work this paper cites.

Approximation by superpositions of a sigmoidal function

G. Cybenko · 1989

Earlier work this paper cites.

Approximation and estimation bounds for artificial neural networks

Andrew R. Barron · 1994

Earlier work this paper cites.

A learning result for continuous-time recurrent neural networks

Eduardo D. Sontag · 1998

Earlier work this paper cites.

A state-space method for language modeling

V. Siivola and A. Honkela · 2003

Earlier work this paper cites.

On a Constructive Proof of Kolmogorov’s Superposition Theorem

Jürgen Braun and Michael Griebel · 2009

Earlier work this paper cites.

Approximationstheorie: Tschebyscheffsche Approximation mit Anwendungen

Lothar Collatz and Werner Krabs · 2013

Cited alongside, same era.

The Power of Depth for Feedforward Neural Networks

Ronen Eldan and Ohad Shamir · 2016

Cited alongside, same era.

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017

Cited alongside, same era.

An empirical evaluation of generic convolutional and recurrent networks for sequence modeling

Original

Shaojie Bai, J Zico Kolter, and Vladlen Koltun · 2018

Cited alongside, same era.

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

Eric Martin and Chris Cundy · 2018

Cited alongside, same era.

HiPPO: Recurrent Memory with Optimal Polynomial Projections

Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks

Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li · 2022

Later among the works it cites.

CKConv: Continuous Kernel Convolution For Sequential Data

David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomczak, and Mark Hoogendoorn · 2022

Later among the works it cites.

Simplified State Space Layers for Sequence Modeling

Jimmy T. H. Smith, Andrew Warrington, and Scott Linderman · 2023

Closest in time.

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Re · 2023

Closest in time.

How to Train your HIPPO: State Space Models with Generalized Orthogonal Basis Projections

Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Re · 2023

Closest in time.

Hyena Hierarchy: Towards Larger Convolutional Language Models