The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers

Understand

We propose a new framework for reasoning about generalization in deep learning.

• The core idea is to couple the Real World, where optimizers take stochastic gradient steps on the empirical loss, to an Ideal World, where optimizers take steps on the population loss.
• This leads to an alternate decomposition of test error into: (1) the Ideal World test error plus (2) the gap between the two worlds.
• If the gap (2) is universally small, this reduces the problem of generalization in offline learning to the problem of optimization in online learning.