Semantic Composition in Visually Grounded Language Models

Understand

What is sentence meaning and its ideal representation? Much of the expressive power of human language derives from semantic composition, the mind's ability to represent meaning hierarchically & relationally over constituents.

• At the same time, much sentential meaning is outside the text and requires grounding in sensory, motor, and experiential modalities to be adequately learned.
• Although large language models display considerable compositional ability, recent work shows that visually-grounded language models drastically fail to represent compositional structure.
• In this thesis, we explore whether & how models compose visually grounded semantics, and how we might improve their ability to do so.