MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

Understand

Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

• Although widely adopted, transformers require scaling memory and compute linearly during inference.
• A recent stream of work linearized the softmax operation, resulting in powerful recurrent neural network (RNN) models with constant memory and compute costs such as DeltaNet, Mamba or xLSTM.
• These models can be unified by noting that their recurrent layer dynamics can all be derived from an in-context regression objective, approximately optimized through an online learning rule.