Gradient Descent Quantizes ReLU Network Features

Understand

Deep neural networks are often trained in the over-parametrized regime (i.e.

• with far more parameters than training examples), and understanding why the training converges to solutions that generalize remains an open problem.
• Several studies have highlighted the fact that the training procedure, i.e.
• mini-batch Stochastic Gradient Descent (SGD) leads to solutions that have specific properties in the loss landscape.