Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise

Understand

Increasing the size of overparameterized neural networks has been a key in achieving state-of-the-art performance.

• This is captured by the double descent phenomenon, where the test loss follows a decreasing-increasing-decreasing pattern (or sometimes monotonically decreasing) as model width increases.
• However, the effect of label noise on the test loss curve has not been fully explored.
• In this work, we uncover an intriguing phenomenon where label noise leads to a \textit{final ascent} in the originally observed double descent curve.