Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

Understand

Conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets.

• However, as many real-world scenarios involve interaction among multiple agents, it is important to resolve offline RL in the multi-agent setting.
• Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that offline RL algorithms will also transfer to multi-agent settings directly.
• Surprisingly, we empirically observe that conservative offline RL algorithms do not work well in the multi-agent setting -- the performance degrades significantly with an increasing number of agents.