Visual representations in the human brain are aligned with large language models

Understand

The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment.

• However, a quantitative approach for studying this information remains elusive.
• Here, we test whether the contextual information encoded in large language models (LLMs) is beneficial for modelling the complex visual information extracted by the brain from natural scenes.
• We show that LLM embeddings of scene captions successfully characterise brain activity evoked by viewing the natural scenes.