Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data

Understand

Discrete diffusion models with absorbing processes have shown promise in language modeling.

• The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score.
• In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form.
• Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities.