MADiff: Offline Multi-agent Learning with Diffusion Models

Understand

Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task.

• Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness.
• Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear.
• Generating trajectories for each agent with independent DMs may impede coordination, while concatenating all agents' information can lead to low sample efficiency.