Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications

Understand

Deep neural networks with remarkably strong generalization performances are usually over-parameterized.

• Despite explicit regularization strategies are used for practitioners to avoid over-fitting, the impacts are often small.
• Some theoretical studies have analyzed the implicit regularization effect of stochastic gradient descent (SGD) on simple machine learning models with certain assumptions.
• However, how it behaves practically in state-of-the-art models and real-world datasets is still unknown.