PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

Understand

Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization.

• Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations.
• Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random.
• We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust.