DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Understand

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task.

• Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance.
• This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics.
• We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model.