Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Understand

State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling.

• Moreover, despite scaling nearly linearly in sequence length instead of quadratically, SSMs are still slower than Transformers due to poor hardware utilization.
• In this paper, we make progress on understanding the expressivity gap between SSMs and attention in language modeling, and on reducing the hardware barrier between SSMs and attention.
• First, we use synthetic language modeling tasks to understand the gap between SSMs and attention.