Self-Adaptive Training: beyond Empirical Risk Minimization

Understand

We propose self-adaptive training---a new training algorithm that dynamically corrects problematic training labels by model predictions without incurring extra computational cost---to improve generalization of deep learning for potentially corrupted training data.

• This problem is crucial towards robustly learning from data that are corrupted by, e.g., label noises and out-of-distribution samples.
• The standard empirical risk minimization (ERM) for such data, however, may easily overfit noises and thus suffers from sub-optimal performance.
• In this paper, we observe that model predictions can substantially benefit the training process: self-adaptive training significantly improves generalization over ERM under various levels of noises, and mitigates the overfitting issue in both natural and adversarial training.