Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat

Understand

We propose a grounded dialogue state encoder which addresses a foundational issue on how to integrate visual grounding with dialogue system components.

• As a test-bed, we focus on the GuessWhat?! game, a two-player game where the goal is to identify an object in a complex visual scene by asking a sequence of yes/no questions.
• Our visually-grounded encoder leverages synergies between guessing and asking questions, as it is trained jointly using multi-task learning.
• We further enrich our model via a cooperative learning regime.