Self-Imitation Advantage Learning

Understand

Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems.

• It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks.
• Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward.
• We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning.