More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory

Understand

In our era of enormous neural networks, empirical progress has been driven by the philosophy that more is better.

• Recent deep learning practice has found repeatedly that larger model size, more data, and more computation (resulting in lower training loss) improves performance.
• In this paper, we give theoretical backing to these empirical observations by showing that these three properties hold in random feature (RF) regression, a class of models equivalent to shallow networks with only the last layer trained.
• Concretely, we first show that the test risk of RF regression decreases monotonically with both the number of features and the number of samples, provided the ridge penalty is tuned optimally.