Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

Understand

The training process of ReLU neural networks often exhibits complicated nonlinear phenomena.

• The nonlinearity of models and non-convexity of loss pose significant challenges for theoretical analysis.
• Therefore, most previous theoretical works on the optimization dynamics of neural networks focus either on local analysis (like the end of training) or approximate linear models (like Neural Tangent Kernel).
• In this work, we conduct a complete theoretical characterization of the training process of a two-layer ReLU network trained by Gradient Flow on a linearly separable data.