Proximal Policy Gradient: PPO with Policy Gradient

Understand

In this paper, we propose a new algorithm PPG (Proximal Policy Gradient), which is close to both VPG (vanilla policy gradient) and PPO (proximal policy optimization).

• The PPG objective is a partial variation of the VPG objective and the gradient of the PPG objective is exactly same as the gradient of the VPG objective.
• To increase the number of policy update iterations, we introduce the advantage-policy plane and design a new clipping strategy.
• We perform experiments in OpenAI Gym and Bullet robotics environments for ten random seeds.