Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control

Understand

Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs.

• Such capabilities are difficult to learn solely from task-specific data.
• This has led to the emergence of pre-trained vision-language models as a tool for transferring representations learned from internet-scale data to downstream tasks and new domains.
• However, commonly used contrastively trained representations such as in CLIP have been shown to fail at enabling embodied agents to gain a sufficiently fine-grained scene understanding -- a capability vital for control.