Generalization bounds for deep learning

Understand

Generalization in deep learning has been the topic of much recent theoretical and empirical research.

• Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning.
• Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous.
• We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data.