Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms

Understand

Generative pre-training is often framed through a false dichotomy between autoregressive models for discrete signals and diffusion models for continuous signals.

• We argue that the dichotomy is false because it conflates model family, data representation, training objective, and inference procedure.
• Autoregression is an inference procedure that expands a sequence through normalized conditional draws, while diffusion is a refinement procedure that repeatedly revises an existing state.
• The more useful contrast is therefore not autoregressive versus diffusion, but discrete tokens learned with cross-entropy versus continuous tokens learned with diffusion-style objectives, together with the inference algorithms used to sample from them.