In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness

Understand

A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context during inference implicitly through some data, and tasked with making a prediction in that context.

• As such, that learner must adapt to the context without additional training.
• We explore the role of softmax attention in an ICL setting where each context encodes a regression task.
• We show that an attention unit learns a window that it uses to implement a nearest-neighbors predictor adapted to the landscape of the pretraining tasks.