Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?

Understand

An important assumption that comes with using LLMs on psycholinguistic data has gone unverified.

• LLM-based predictions are based on subword tokenization, not decomposition of words into morphemes.
• Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data.
• Our results replicate previous findings and provide evidence that in the aggregate, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation.