The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

Understand

Large language models have the ability to generate text that mimics patterns in their inputs.

• We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning (ICL) capability emerges.
• In our setting, each example is sampled from a Markov chain drawn from a prior distribution over Markov chains.
• Transformers trained on this task form \emph{statistical induction heads} which compute accurate next-token probabilities given the bigram statistics of the context.