Failures of Gradient-Based Deep Learning

Understand

In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art.

• However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms.
• We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties.
• We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.