Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Understand

Self-attention performs well in long context but has quadratic complexity.

• Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states.
• We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states.
• The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning.