Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models

Understand

When solving challenging problems, language models (LMs) are able to identify relevant information from long and complicated contexts.

• To study how LMs solve retrieval tasks in diverse situations, we introduce ORION, a collection of structured retrieval tasks spanning six domains, from text understanding to coding.
• Each task in ORION can be represented abstractly by a request (e.g.
• a question) that retrieves an attribute (e.g.