Mimetic Initialization Helps State Space Models Learn to Recall

Understand

Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with respect to their input sequence length.

• But in practice, state space models have fairly large state sizes, and we conjecture that they should be able to perform much better at these tasks than previously reported.
• We investigate whether their poor copying and recall performance could be due in part to training difficulties rather than fundamental capacity constraints.
• Based on observations of their "attention" maps, we propose a structured initialization technique that allows state space layers to more readily mimic attention.