2022

SinFusion: Training Diffusion Models on a Single Image or Video

Nikankin, Yaniv, Haim, Niv, Irani, Michal

Understand

Diffusion models exhibited tremendous progress in image and video generation, exceeding GANs in quality and diversity.

However, they are usually trained on very large datasets and are not naturally adapted to manipulate a given input image or video.
In this paper we show how this can be resolved by training a diffusion model on a single input image or video.
Our image/video-specific diffusion model (SinFusion) learns the appearance and dynamics of the single image or video, while utilizing the conditioning capabilities of diffusion models.

Built on

On tables of random numbers
Kolmogorov, A. N · 1963
Earlier work this paper cites.
Texture synthesis by non-parametric sampling
Efros, A. A. and Leung, T. K · 1999
Earlier work this paper cites.
Image quilting for texture synthesis and transfer
Efros, A. A. and Freeman, W. T · 2001
Earlier work this paper cites.
Zlib compression library
Gailly, J.-l. and Adler, M · 2004
Earlier work this paper cites.
Summarizing visual data using bidirectional similarity
Simakov, D., Caspi, Y., Shechtman, E., and Irani, M · 2008
Earlier work this paper cites.
Using cloud shadows to infer scene structure and camera calibration
Jacobs, N., Bies, B., and Pless, R · 2010
Earlier work this paper cites.
A database and evaluation methodology for optical flow
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M. J., and Szeliski, R · 2011
Earlier work this paper cites.
Two cloud-based cues for estimating scene structure and camera calibration
Jacobs, N., Abrams, A., and Pless, R · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Delving deeper into convolutional networks for learning video representations
Original
Ballas, N., Yao, L., Pal, C., and Courville, A · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Original
Radford, A., Metz, L., and Chintala, S · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Generating videos with scene dynamics
Original
Vondrick, C., Pirsiavash, H., and Torralba, A · 2016
Earlier work this paper cites.
Stochastic variational video prediction
Original
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S · 2017
Earlier work this paper cites.
Progressive growing of gans for improved quality, stability, and variation
Original
Karras, T., Aila, T., Laine, S., and Lehtinen, J · 2017
Earlier work this paper cites.
Automatic differentiation in pytorch
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A · 2017
Earlier work this paper cites.
Decomposing motion and content for natural video sequence prediction
Original
Villegas, R., Yang, J., Hong, S., Lin, X., and Lee, H · 2017
Earlier work this paper cites.
Recycle-gan: Unsupervised video retargeting
Bansal, A., Ma, S., Ramanan, D., and Sheikh, Y · 2018
Earlier work this paper cites.
Large scale gan training for high fidelity natural image synthesis
Original
Brock, A., Donahue, J., and Simonyan, K · 2018
Earlier work this paper cites.

Similar

Stochastic video generation with a learned prior
Denton, E. and Fergus, R · 2018
Cited alongside, same era.
Ingan: Capturing and remapping the” dna” of a natural image
Original
Shocher, A., Bagon, S., Isola, P., and Irani, M · 2018
Cited alongside, same era.
Mocogan: Decomposing motion and content for video generation
Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J · 2018
Cited alongside, same era.
Video-to-video synthesis
Original
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., and Catanzaro, B · 2018
Cited alongside, same era.
Adversarial video generation on complex datasets
Original
Clark, A., Donahue, J., and Simonyan, K · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Karras, T., Laine, S., and Aila, T · 2019
Cited alongside, same era.
Singan: Learning a generative model from a single natural image
Shaham, T. R., Dekel, T., and Michaeli, T · 2019
Cited alongside, same era.
Generative modeling by estimating gradients of the data distribution
Song, Y. and Ermon, S · 2019
Cited alongside, same era.
Self-attention generative adversarial networks
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A · 2019
Cited alongside, same era.
Hierarchical patch vae-gan: Generating diverse videos from a single sample
Original
Gur, S., Benaim, S., and Wolf, L · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Ho, J., Jain, A., and Abbeel, P · 2020
Cited alongside, same era.
Adversarial score matching and improved sampling for image generation
Original
Jolicoeur-Martineau, A., Piché-Taillefer, R., Combes, R. T. d., and Mitliagkas, I · 2020
Cited alongside, same era.

Then

Improved denoising diffusion probabilistic models
Nichol, A. Q. and Dhariwal, P · 2021
Later among the works it cites.
Blended diffusion for text-driven editing of natural images
Avrahami, O., Lischinski, D., and Fried, O · 2022
Closest in time.
A survey on generative diffusion model
Original
Cao, H., Tan, C., Gao, Z., Chen, G., Heng, P.-A., and Li, S. Z · 2022
Closest in time.
Diffusion models in vision: A survey
Original
Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M · 2022
Closest in time.
An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022
Original
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D · 2022
Closest in time.
Drop the gan: In defense of patches nearest neighbors as single image generative models
Granot, N., Feinstein, B., Shocher, A., Bagon, S., and Irani, M · 2022
Closest in time.
Flexible diffusion modeling of long videos
Original
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, F · 2022
Closest in time.
Diffusion models for video prediction and infilling
Original
Höppe, T., Mehrjou, A., Bauer, S., Nielsen, D., and Dittadi, A · 2022
Closest in time.
Imagic: Text-based real image editing with diffusion models
Original
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M · 2022
Closest in time.
Sinddm: A single image denoising diffusion model
Original
Kulikov, V., Yadin, S., Kleiner, M., and Michaeli, T · 2022
Closest in time.
A convnet for the 2020s
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S · 2022
Closest in time.
High-resolution image synthesis with latent diffusion models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B · 2022
Closest in time.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K · 2022
Closest in time.
Make-a-video: Text-to-video generation without text-video data
Original
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al · 2022
Closest in time.
Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2
Skorokhodov, I., Tulyakov, S., and Elhoseiny, M · 2022
Closest in time.
Unitune: Text-driven image editing by fine tuning an image generation model on a single image
Original
Valevski, D., Kalman, M., Matias, Y., and Leviathan, Y · 2022
Closest in time.
Phenaki: Variable length video generation from open domain textual description
Original
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D · 2022
Closest in time.
Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation
Original
Voleti, V., Jolicoeur-Martineau, A., and Pal, C · 2022
Closest in time.
Sindiffusion: Learning a diffusion model from a single natural image
Original
Wang, W., Bao, J., Zhou, W., Chen, D., Chen, D., Yuan, L., and Li, H · 2022
Closest in time.
Gan inversion: A survey
Xia, W., Zhang, Y., Yang, Y., Xue, J.-H., Zhou, B., and Yang, M.-H · 2022
Closest in time.
Diffusion probabilistic modeling for video generation
Original
Yang, R., Srivastava, P., and Mandt, S · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…