DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models

Understand

We present DiffusionBERT, a new generative masked language model based on discrete diffusion models.

• Diffusion models and many pre-trained language models have a shared training objective, i.e., denoising, making it possible to combine the two powerful models and enjoy the best of both worlds.
• On the one hand, diffusion models offer a promising training strategy that helps improve the generation quality.
• On the other hand, pre-trained denoising language models (e.g., BERT) can be used as a good initialization that accelerates convergence.