2022

gDDIM: Generalized denoising diffusion implicit models

Zhang, Qinsheng, Tao, Molei, Chen, Yongxin

Understand

Our goal is to extend the denoising diffusion implicit model (DDIM) to general diffusion models~(DMs) besides isotropic diffusions.

Instead of constructing a non-Markov noising process as in the original DDIM, we examine the mechanism of DDIM from a numerical perspective.
We discover that the DDIM can be obtained by using some specific approximations of the score when solving the corresponding stochastic differential equation.
We present an interpretation of the accelerating effects of DDIM that also explains the advantages of a deterministic sampling scheme over the stochastic one for fast sampling.

Built on

Reverse-time diffusion equation models
Brian D.O. Anderson · 1982
Earlier work this paper cites.
Nonlinear dimensionality reduction by locally linear embedding
Sam T. Roweis and Lawrence K. Saul · 2000
Earlier work this paper cites.
A global geometric framework for nonlinear dimensionality reduction
Joshua B. Tenenbaum, Vin De Silva, and John C. Langford · 2000
Earlier work this paper cites.
Numerical analysis, 2005
Timothy Sauer · 2005
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2006
Earlier work this paper cites.
Numerical recipes 3rd edition: The art of scientific computing
William H Press, Saul A Teukolsky, William T Vetterling, and Brian P Flannery · 2007
Earlier work this paper cites.
Exponential integrators
Marlis Hochbruck and Alexander Ostermann · 2010
Earlier work this paper cites.
Denoising Diffusion Implicit Models
Original
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2010
Earlier work this paper cites.
Score-Based Generative Modeling through Stochastic Differential Equations
Original
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2011
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
Pascal Vincent · 2011
Earlier work this paper cites.
Interpretation and generalization of score matching
Original
Siwei Lyu · 2012
Earlier work this paper cites.
Stochastic differential equations: an introduction with applications
Bernt Oksendal · 2013
Earlier work this paper cites.
Exponential time-differencing with embedded Runge–Kutta adaptive step control
P. Whalen, M. Brio, and J.V. Moloney · 2014
Earlier work this paper cites.

Similar

Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Cited alongside, same era.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Cited alongside, same era.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud · 2018
Cited alongside, same era.
Applied stochastic differential equations , volume 10
Simo Särkkä and Arno Solin · 2019
Cited alongside, same era.
Generative modeling by estimating gradients of the data distribution
Yang Song and Stefano Ermon · 2019
Cited alongside, same era.
Fast sampling of diffusion models with exponential integrator
Original
Qinsheng Zhang and Yongxin Chen · 2020
Cited alongside, same era.
Diffusion Models Beat GANs on Image Synthesis
Original
Prafulla Dhariwal and Alex Nichol · 2021
Cited alongside, same era.
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
Original
Tim Dockhorn, Arash Vahdat, and Karsten Kreis · 2021
Cited alongside, same era.
Snips: Solving noisy inverse problems stochastically
Bahjat Kawar, Gregory Vaksman, and Michael Elad · 2021
Cited alongside, same era.
Knowledge distillation in iterative generative models for improved sampling speed
Original
Eric Luhman and Troy Luhman · 2021
Cited alongside, same era.
Improved denoising diffusion probabilistic models
Original
Alex Nichol and Prafulla Dhariwal · 2021
Cited alongside, same era.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Cited alongside, same era.

Then

Maximum likelihood training of score-based diffusion models
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon · 2021
Later among the works it cites.
Score-based generative modeling in latent space
Arash Vahdat, Karsten Kreis, and Jan Kautz · 2021
Later among the works it cites.
Learning to efficiently sample from diffusion probabilistic models
Original
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan · 2021
Later among the works it cites.
Tackling the generative learning trilemma with denoising diffusion gans
Original
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat · 2021
Later among the works it cites.
Diffusion normalizing flow
Qinsheng Zhang and Yongxin Chen · 2021
Later among the works it cites.
Analytic-DPM: An Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models
Original
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang · 2022
Closest in time.
Blurring diffusion models
Original
Emiel Hoogeboom and Tim Salimans · 2022
Closest in time.
Elucidating the design space of diffusion-based generative models
Original
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine · 2022
Closest in time.
Pseudo Numerical Methods for Diffusion Models on Manifolds
Original
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao · 2022
Closest in time.
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps
Original
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu · 2022
Closest in time.
Generative modelling with inverse heat dissipation
Original
Severi Rissanen, Markus Heinonen, and Arno Solin · 2022
Closest in time.
Progressive Distillation for Fast Sampling of Diffusion Models
Original
Tim Salimans and Jonathan Ho · 2022
Closest in time.
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
Original
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…