$\textit{New News}$: System-2 Fine-tuning for Robust Integration of New Knowledge

Understand

Humans and intelligent animals can internalize new information and accurately internalize their implications to perform downstream tasks.

• While large language models (LLMs) can achieve this through in-context learning (ICL) when the information (news) is explicitly given as context, adequately integrating the information into model weights via fine-tuning remains challenging.
• In this paper, we introduce New News, a dataset composed of hypothetical yet plausible news spanning multiple domains (mathematics, coding, discoveries, leaderboards, events), accompanied by downstream evaluation questions whose correct answers critically depend on understanding and internalizing the news.
• First, we demonstrate a substantial gap between naive fine-tuning and in-context learning (FT-ICL gap) on our dataset.