Language Models Represent Space and Time

Understand

The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world.

• We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models.
• We discover that LLMs learn linear representations of space and time across multiple scales.
• These representations are robust to prompting variations and unified across different entity types (e.g.