Efficient Adversarial Training with Robust Early-Bird Tickets

Understand

Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs).

• However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent.
• Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs), far before parameters converge.
• Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time.