Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization

Understand

Adversarial Imitation Learning alternates between learning a discriminator -- which tells apart expert's demonstrations from generated ones -- and a generator's policy to produce trajectories that can fool this discriminator.

• This alternated optimization is known to be delicate in practice since it compounds unstable adversarial training with brittle and sample-inefficient reinforcement learning.
• We propose to remove the burden of the policy optimization steps by leveraging a novel discriminator formulation.
• Specifically, our discriminator is explicitly conditioned on two policies: the one from the previous generator's iteration and a learnable policy.