Extensible Embedding: A Flexible Multipler For LLM's Context Length

Understand

Large language models (LLMs) call for extension of context to handle many critical applications.

• However, the existing approaches are prone to expensive costs and inferior quality of context extension.
• In this work, we propose Extensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness.
• Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token.