Adversarial Training Versus Weight Decay

Understand

Performance-critical machine learning models should be robust to input perturbations not seen during training.

• Adversarial training is a method for improving a model's robustness to some perturbations by including them in the training process, but this tends to exacerbate other vulnerabilities of the model.
• The adversarial training framework has the effect of translating the data with respect to the cost function, while weight decay has a scaling effect.
• Although weight decay could be considered a crude regularization technique, it appears superior to adversarial training as it remains stable over a broader range of regimes and reduces all generalization errors.