LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation

Understand

Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches.

• Motivated by the efficiency gains of hybrid models and the broad availability of pretrained large transformer backbones, we explore transitioning transformer models into hybrid architectures for a more efficient generation.
• In this work, we propose LightTransfer, a lightweight method that transforms models such as LLaMA into hybrid variants.
• Our approach identifies lazy layers -- those focusing on recent or initial tokens -- and replaces their full attention with streaming attention.