Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization

Understand

The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it.

• Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in expectation.
• In this paper, we establish various generalization error bounds with probabilistic guarantee for the SGD.
• Specifically, for both general nonconvex loss functions and gradient dominant loss functions, we characterize the on-average stability of the iterates generated by SGD in terms of the on-average variance of the stochastic gradients.