Learning Non-Vacuous Generalization Bounds from Optimization

Understand

One of the fundamental challenges in the deep learning community is to theoretically understand how well a deep neural network generalizes to unseen data.

• However, current approaches often yield generalization bounds that are either too loose to be informative of the true generalization error or only valid to the compressed nets.
• In this study, we present a simple yet non-vacuous generalization bound from the optimization perspective.
• We achieve this goal by leveraging that the hypothesis set accessed by stochastic gradient algorithms is essentially fractal-like and thus can derive a tighter bound over the algorithm-dependent Rademacher complexity.