A Systematic Analysis of Hybrid Linear Attention

Understand

Transformers face quadratic complexity and memory issues with long sequences, prompting the adoption of linear attention mechanisms using fixed-size hidden states.

• However, linear models often suffer from limited recall performance, leading to hybrid architectures that combine linear and full attention layers.
• Despite extensive hybrid architecture research, the choice of linear attention component has not been deeply explored.
• We systematically evaluate various linear attention models across generations - vector recurrences to advanced gating mechanisms - both standalone and hybridized.