Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

Understand

Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity.

• Indeed, neural networks often have many more parameters than there are data points, yet still provide good generalization performance.
• Moreover, when we measure generalization as a function of parameters, we see double descent behaviour, where the test error decreases, increases, and then again decreases.
• We show that many of these properties become understandable when viewed through the lens of effective dimensionality, which measures the dimensionality of the parameter space determined by the data.