Attention, not scale, drives human-AI alignment in multimodal language prediction

Understand

Humans routinely draw on visual context to predict upcoming words.

• To what extent current vision-language models produce comparable behaviour is unclear.
• Here we placed five state-of-the-art pretrained systems side-by-side with 600 human participants in a web-based Visual-World Paradigm.
• On each of 100 six-second movie clips, models and participants received either text only or synchronised video and text and judged how likely a specified target word was to appear next; human eye movements were tracked throughout.