ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Understand

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens.

• While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective.
• As an alternative, we propose a more sample-efficient pre-training task called replaced token detection.
• Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network.