Analyzing and Improving the Training Dynamics of Diffusion Models

Understand

Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets.

• In this paper, we identify and rectify several causes for uneven and ineffective training in the popular ADM diffusion model architecture, without altering its high-level structure.
• Observing uncontrolled magnitude changes and imbalances in both the network activations and weights over the course of training, we redesign the network layers to preserve activation, weight, and update magnitudes on expectation.
• We find that systematic application of this philosophy eliminates the observed drifts and imbalances, resulting in considerably better networks at equal computational complexity.