An Explanation of In-context Learning as Implicit Bayesian Inference

Understand

Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples.

• The LM learns from these examples without being explicitly pretrained to learn.
• Thus, it is unclear what enables in-context learning.
• In this paper, we study how in-context learning can emerge when pretraining documents have long-range coherence.