Focused Transformer: Contrastive Training for Context Scaling

Understand

Large language models have an exceptional capability to incorporate new information in a contextual manner.

• However, the full potential of such an approach is often restrained due to a limitation in the effective context length.
• One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs.
• Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys.