Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

Understand

We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models.

• The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits.
• To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables.
• We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality.