Data Distributional Properties Drive Emergent In-Context Learning in Transformers

Understand

Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it.

• This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself.
• In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes.
• In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed.