On the Power of Context-Enhanced Learning in LLMs

Understand

We formalize a new concept for LLMs, context-enhanced learning.

• It involves standard gradient-based learning on text except that the context is enhanced with additional data on which no auto-regressive gradients are computed.
• This setting is a gradient-based analog of usual in-context learning (ICL) and appears in some recent works.
• Using a multi-step reasoning task, we prove in a simplified setting that context-enhanced learning can be exponentially more sample-efficient than standard learning when the model is capable of ICL.