In-Context Learning with Long-Context Models: An In-Depth Exploration

Understand

As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets.

• We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models.
• We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations.
• We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data.