2022

Diffusion Models in Vision: A Survey

Croitoru, Florinel-Alin, Hondru, Vlad, Ionescu, Radu Tudor et al.

Understand

Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling.

A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage.
In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise.
In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step.

Built on

W. Feller, “On the Theory of Stochastic Processes, with Particular Reference to Applications,” in
1949
Earlier work this paper cites.
K. Miyasawa, “An empirical Bayes estimator of the mean of a normal population,”
1961
Earlier work this paper cites.
B. D. Anderson, “Reverse-time diffusion equation models,”
1982
Earlier work this paper cites.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,”
2006
Earlier work this paper cites.
Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. J. Huang, “A tutorial on energy-based learning,” in
2006
Earlier work this paper cites.
J. Ngiam, Z. Chen, P. W. Koh, and A. Y. Ng, “Learning Deep Energy Models,” in
2011
Earlier work this paper cites.
P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,”
2011
Earlier work this paper cites.
K. Swersky, M. Ranzato, D. Buchman, B. M. Marlin, and N. Freitas, “On Autoencoders and Score Matching for Energy Based Models,” in
2011
Earlier work this paper cites.
Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,”
2013
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in
2014
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in
2014
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using non-equilibrium thermodynamics,” in
2015
Earlier work this paper cites.
L. Dinh, D. Krueger, and Y. Bengio, “NICE: Non-linear Independent Components Estimation,” in
2015
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in
2015
Earlier work this paper cites.
MIT Press, 2016
I. Goodfellow, Y. Bengio, and A. Courville, · 2016
Earlier work this paper cites.
A. van den Oord, N. Kalchbrenner, L. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves, “Conditional Image Generation with PixelCNN Decoders,” in
2016
Earlier work this paper cites.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved Techniques for Training GANs,” in
2016
Earlier work this paper cites.
A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” in
2016
Earlier work this paper cites.
Y. Marzouk, T. Moselhy, M. Parno, and A. Spantini, “Sampling via Measure Transport: An Introduction,” in
2016
Earlier work this paper cites.
Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation,” in
2016
Earlier work this paper cites.
I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, “beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” in
2017
Earlier work this paper cites.
S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” in
2017
Earlier work this paper cites.
A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in
2017
Earlier work this paper cites.
L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using Real NVP,” in
2017
Earlier work this paper cites.
T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma, “PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications,” in
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in
2017
Earlier work this paper cites.
M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” in
2017
Earlier work this paper cites.
C. K. Sønderby, J. Caballero, L. Theis, W. Shi, and F. Huszár, “Amortised map inference for image super-resolution,” in
2017
Earlier work this paper cites.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in
2017
Earlier work this paper cites.
A. Van Den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,”
2017
Earlier work this paper cites.
G. Lin, A. Milan, C. Shen, and I. Reid, “RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation,” in
2017
Earlier work this paper cites.
R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” in
2017
Earlier work this paper cites.
A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,”
Original
2018
Earlier work this paper cites.
X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” in
2018
Earlier work this paper cites.
G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca, “An unsupervised learning model for deformable medical image registration,” in
2018
Earlier work this paper cites.
Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” in
2019
Earlier work this paper cites.
Y. Song, S. Garg, J. Shi, and S. Ermon, “Sliced Score Matching: A Scalable Approach to Density and Score Estimation,” in
2019
Earlier work this paper cites.
N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in
2019
Earlier work this paper cites.
R. T. Ionescu, F. S. Khan, M.-I. Georgescu, and L. Shao, “Object-Centric Auto-Encoders and Dummy Anomalies for Abnormal Event Detection in Video,” in
2019
Earlier work this paper cites.
T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in
2019
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in
2020
Earlier work this paper cites.
Y. Song and S. Ermon, “Improved techniques for training score-based generative models,” in
2020
Earlier work this paper cites.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” in
2020
Earlier work this paper cites.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in
2020
Earlier work this paper cites.
Y. Shen, J. Gu, X. Tang, and B. Zhou, “Interpreting the Latent Space of GANs for Semantic Face Editing,” in
2020
Earlier work this paper cites.
M. Shannon, B. Poole, S. Mariooryad, T. Bagby, E. Battenberg, D. Kao, D. Stanton, and R. Skerry-Ryan, “Non-saturating GAN training as divergence minimization,”
Original
2020
Earlier work this paper cites.
J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients – How easy is it to break privacy in federated learning?,” in
2020
Earlier work this paper cites.
N. Kovachki, R. Baptista, B. Hosseini, and Y. Marzouk, “Conditional sampling with monotone GANs,”
Original
2020
Earlier work this paper cites.
I. Oguz, J. D. Malone, Y. Atay, and Y. K. Tao, “Self-fusion for OCT noise reduction,” in
2020
Earlier work this paper cites.
X. Li, T.-K. L. Wong, R. T. Chen, and D. Duvenaud, “Scalable gradients for stochastic differential equations,” in
2020
Earlier work this paper cites.
Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” in
2021
Earlier work this paper cites.
P. Dhariwal and A. Nichol, “Diffusion models beat GANs on image synthesis,” in
2021
Earlier work this paper cites.
A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in
2021
Earlier work this paper cites.
J. Song, C. Meng, and S. Ermon, “Denoising Diffusion Implicit Models,” in
2021
Earlier work this paper cites.
D. Watson, W. Chan, J. Ho, and M. Norouzi, “Learning fast samplers for diffusion models by differentiating through sample quality,” in
2021
Earlier work this paper cites.
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,” in
2021
Earlier work this paper cites.

Similar

Y. Song, C. Durkan, I. Murray, and S. Ermon, “Maximum likelihood training of score-based diffusion models,” in
2021
Cited alongside, same era.
A. Sinha, J. Song, C. Meng, and S. Ermon, “D2C: Diffusion-decoding models for few-shot conditional generation,” in
2021
Cited alongside, same era.
A. Vahdat, K. Kreis, and J. Kautz, “Score-based generative modeling in latent space,” in
2021
Cited alongside, same era.
C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,”
Original
2021
Cited alongside, same era.
K. Pandey, A. Mukherjee, P. Rai, and A. Kumar, “VAEs meet diffusion models: Efficient and high-fidelity generation,” in
2021
Cited alongside, same era.
G. Batzolis, J. Stanczuk, C.-B. Schönlieb, and C. Etmann, “Conditional image generation with score-based diffusion models,”
Original
2021
Cited alongside, same era.
M. Daniels, T. Maunu, and P. Hand, “Score-based generative neural networks for large-scale optimal transport,” in
2021
Cited alongside, same era.
P. Esser, R. Rombach, A. Blattmann, and B. Ommer, “ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis,” in
2021
Cited alongside, same era.
J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon, “ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models,” in
2021
Cited alongside, same era.
C. Meng, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations,” in
2021
Cited alongside, same era.
T. Amit, E. Nachmani, T. Shaharbany, and L. Wolf, “SegDiff: Image Segmentation with Diffusion Probabilistic Models,”
Original
2021
Cited alongside, same era.
R. S. Zimmermann, L. Schott, Y. Song, B. A. Dunn, and D. A. Klindt, “Score-based generative classifiers,” in
2021
Cited alongside, same era.

Then

J. Wolleb, R. Sandkühler, F. Bieder, and P. C. Cattin, “The Swiss Army Knife for Image-to-Image Translation: Multi-Task Diffusion Models,”
Original
2022
Closest in time.
D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko, “Label-Efficient Semantic Segmentation with Diffusion Models,” in
2022
Closest in time.
A. Graikos, N. Malkin, N. Jojic, and D. Samaras, “Diffusion models as plug-and-play priors,”
Original
2022
Closest in time.
J. Wolleb, R. Sandkühler, F. Bieder, P. Valmaggia, and P. C. Cattin, “Diffusion Models for Implicit Image Segmentation Ensembles,” in
2022
Closest in time.
W. H. Pinaya, M. S. Graham, R. Gray, P. F. Da Costa, P.-D. Tudosiu, P. Wright, Y. H. Mah, A. D. MacKinnon, J. T. Teo, R. Jager,
Original
2022
Closest in time.
J. Wolleb, F. Bieder, R. Sandkühler, and P. C. Cattin, “Diffusion Models for Medical Anomaly Detection,”
Original
2022
Closest in time.
J. Wyatt, A. Leach, S. M. Schmon, and C. G. Willcocks, “AnoDDPM: Anomaly Detection With Denoising Diffusion Probabilistic Models Using Simplex Noise,” in
2022
Closest in time.
F.-A. Croitoru, D.-N. Grigore, and R. T. Ionescu, “Discriminability-enforcing loss to improve representation learning,” in
2022
Closest in time.
Y. Benny and L. Wolf, “Dynamic Dual-Output Diffusion Models,” in
2022
Closest in time.
S. Bond-Taylor, P. Hessey, H. Sasaki, T. P. Breckon, and C. G. Willcocks, “Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes,” in
2022
Closest in time.
J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and S. Yoon, “Perception Prioritized Training of Diffusion Models,” in
2022
Closest in time.
K. Deja, A. Kuzina, T. Trzciński, and J. M. Tomczak, “On Analyzing Generative and Denoising Capabilities of Diffusion-Based Deep Generative Models,”
Original
2022
Closest in time.
D. Kim, B. Na, S. J. Kwon, D. Lee, W. Kang, and I.-C. Moon, “Maximum Likelihood Training of Implicit Nonlinear Diffusion Models,”
Original
2022
Closest in time.
L. Liu, Y. Ren, Z. Lin, and Z. Zhao, “Pseudo Numerical Methods for Diffusion Models on Manifolds,” in
2022
Closest in time.
H. Ma, L. Zhang, X. Zhu, J. Zhang, and J. Feng, “Accelerating Score-Based Generative Models for High-Resolution Image Synthesis,”
Original
2022
Closest in time.
V. Sehwag, C. Hazirbas, A. Gordo, F. Ozgenel, and C. Canton, “Generating High Fidelity Data from Low-Density Regions Using Diffusion Models,” in
2022
Closest in time.
Z. Wang, H. Zheng, P. He, W. Chen, and M. Zhou, “Diffusion-GAN: Training GANs with Diffusion,”
Original
2022
Closest in time.
Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the generative learning trilemma with denoising diffusion GANs,” in
2022
Closest in time.
H. Zheng, P. He, W. Chen, and M. Zhou, “Truncated diffusion probabilistic models,”
Original
2022
Closest in time.
F. Bordes, R. Balestriero, and P. Vincent, “High fidelity visualization of what your self-supervised representation knows about,”
2022
Closest in time.
A. Campbell, J. Benton, V. De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet, “A Continuous Time Framework for Discrete Denoising Models,”
Original
2022
Closest in time.
C.-H. Chao, W.-F. Sun, B.-W. Cheng, Y.-C. Lo, C.-C. Chang, Y.-L. Liu, Y.-L. Chang, C.-P. Chen, and C.-Y. Lee, “Denoising Likelihood Score Matching for Conditional Score-Based Data Generation,” in
2022
Closest in time.
J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded Diffusion Models for High Fidelity Image Generation,”
2022
Closest in time.
T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,”
Original
2022
Closest in time.
C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,”
Original
2022
Closest in time.
T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” in
2022
Closest in time.
V. Singh, S. Jandial, A. Chopra, S. Ramesh, B. Krishnamurthy, and V. N. Balasubramanian, “On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models,”
Original
2022
Closest in time.
S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in
2022
Closest in time.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with CLIP latents,”
Original
2022
Closest in time.
Q. Zhang and Y. Chen, “Fast Sampling of Diffusion Models with Exponential Integrator,”
Original
2022
Closest in time.
O. Avrahami, O. Fried, and D. Lischinski, “Blended latent diffusion,”
Original
2022
Closest in time.
G. Batzolis, J. Stanczuk, C.-B. Schönlieb, and C. Etmann, “Non-Uniform Diffusion Models,”
Original
2022
Closest in time.
A. Blattmann, R. Rombach, K. Oktay, and B. Ommer, “Retrieval-Augmented Diffusion Models,”
Original
2022
Closest in time.
M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. N. Suganthan, “Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation,” in
2022
Closest in time.
V. Khrulkov and I. Oseledets, “Understanding DDPM Latent Codes Through Optimal Transport,”
Original
2022
Closest in time.
G. Kim, T. Kwon, and J. C. Ye, “DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation,” in
2022
Closest in time.
Z. Lyu, X. Xu, C. Yang, D. Lin, and B. Dai, “Accelerating Diffusion Models via Early Stop of the Diffusion Process,”
Original
2022
Closest in time.
K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Diffusion Autoencoders: Toward a Meaningful and Decodable Representation,” in
2022
Closest in time.
Y. Shi, V. De Bortoli, G. Deligiannidis, and A. Doucet, “Conditional Simulation Using Diffusion Schrödinger Bridges,” in
2022
Closest in time.
D. Hu, Y. K. Tao, and I. Oguz, “Unsupervised denoising of retinal OCT with diffusion probabilistic model,” in
2022
Closest in time.
H. Chung and J. C. Ye, “Score-based diffusion models for accelerated MRI,”
2022
Closest in time.
M. Özbey, S. U. Dar, H. A. Bedel, O. Dalmaz, Ş. Özturk, A. Güngör, and T. Çukur, “Unsupervised Medical Image Translation with Adversarial Diffusion Models,”
Original
2022
Closest in time.
Y. Song, L. Shen, L. Xing, and S. Ermon, “Solving inverse problems in medical imaging with score-based generative models,” in
2022
Closest in time.
P. Sanchez, A. Kascenas, X. Liu, A. Q. O’Neil, and S. A. Tsaftaris, “What is healthy? generative counterfactual diffusion for lesion localization,” in
2022
Closest in time.
W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood, “Flexible Diffusion Modeling of Long Videos,”
Original
2022
Closest in time.
J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” in
2022
Closest in time.
R. Yang, P. Srivastava, and S. Mandt, “Diffusion Probabilistic Modeling for Video Generation,”
Original
2022
Closest in time.
T. Höppe, A. Mehrjou, S. Bauer, D. Nielsen, and A. Dittadi, “Diffusion Models for Video Prediction and Infilling,”
Original
2022
Closest in time.
G. Giannone, D. Nielsen, and O. Winther, “Few-Shot Diffusion Models,”
Original
2022
Closest in time.
G. Jeanneret, L. Simon, and F. Jurie, “Diffusion Models for Counterfactual Explanations,”
Original
2022
Closest in time.
P. Sanchez and S. A. Tsaftaris, “Diffusion causal models for counterfactual estimation,” in
2022
Closest in time.
O. Özdenizci and R. Legenstein, “Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models,”
Original
2022
Closest in time.
W. Nie, B. Guo, Y. Huang, C. Xiao, A. Vahdat, and A. Anandkumar, “Diffusion models for adversarial purification,” in
2022
Closest in time.
W. Wang, J. Bao, W. Zhou, D. Chen, D. Chen, L. Yuan, and H. Li, “Semantic image synthesis via diffusion models,”
Original
2022
Closest in time.
S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre, “Improving Language Models by Retrieving from Trillions of Tokens,” in
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…