Gated Delta Networks: Improving Mamba2 with Delta Rule

Understand

Linear Transformers have gained attention as efficient alternatives to standard Transformers, but their performance in retrieval and long-context tasks has been limited.

• To address these limitations, recent work has explored two distinct mechanisms: gating for adaptive memory control and the delta update rule for precise memory modifications.
• We observe that these mechanisms are complementary: gating enables rapid memory erasure while the delta rule facilitates targeted updates.
• Building on this insight, we introduce the gated delta rule and develop a parallel training algorithm optimized for modern hardware.