Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Understand

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly.

• Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization.
• In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression.
• We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant.