Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

Understand

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents.

• DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation.
• A large language model orchestrates this autonomous process without requiring human supervision, making it well-suited for lifelong learning scenarios.
• The framework reduces the amount of reward-labeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks.