Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?

Understand

Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions.

• Compared with language, images usually convey information with more details and less ambiguity.
• In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning.
• Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" a stain off a table.