MoM: Linear Sequence Modeling with Mixture-of-Memories

Understand

Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference.

• However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks.
• To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM).
• MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states.