Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models

Understand

The bias-variance trade-off is a central concept in supervised learning.

• In classical statistics, increasing the complexity of a model (e.g., number of parameters) reduces bias but also increases variance.
• Until recently, it was commonly believed that optimal performance is achieved at intermediate model complexities which strike a balance between bias and variance.
• Modern Deep Learning methods flout this dogma, achieving state-of-the-art performance using "over-parameterized models" where the number of fit parameters is large enough to perfectly fit the training data.