The Shattered Gradients Problem: If resnets are the answer, then what is the question?

Understand

A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients.

• Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization.
• In this paper, we identify the shattered gradients problem.
• Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly.