Longhorn: State Space Models are Amortized Online Learners

Understand

Modern large language models are built on sequence modeling via next-token prediction.

• While the Transformer remains the dominant architecture for sequence modeling, its quadratic decoding complexity in sequence length poses a major limitation.
• State-space models (SSMs) present a competitive alternative, offering linear decoding efficiency while maintaining parallelism during training.
• However, most existing SSMs rely on linear recurrence designs that appear somewhat ad hoc.