Learning ReLU Networks via Alternating Minimization

Understand

We propose and analyze a new family of algorithms for training neural networks with ReLU activations.

• Our algorithms are based on the technique of alternating minimization: estimating the activation patterns of each ReLU for all given samples, interleaved with weight updates via a least-squares step.
• The main focus of our paper are 1-hidden layer networks with $k$ hidden neurons and ReLU activation.
• We show that under standard distributional assumptions on the $d-$dimensional input data, our algorithm provably recovers the true `ground truth' parameters in a linearly convergent fashion.