A Cheaper and Better Diffusion Language Model with Soft-Masked Noise

Understand

Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation.

• Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages.
• For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high.
• To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffuse LM, with lower training cost and better performances, inspired by linguistic features in languages.