2023

MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

Bar-Tal, Omer, Yariv, Lior, Lipman, Yaron et al.

Understand

Recent advances in text-to-image generation with diffusion models present transformative capabilities in image quality.

However, user controllability of the generated image, and fast adaptation to new tasks still remains an open challenge, currently mostly addressed by costly and long re-training and fine-tuning or ad-hoc adaptations to specific image generation tasks.
In this work, we present MultiDiffusion, a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning.
At the center of our approach is a new generation process, based on an optimization task that binds together multiple diffusion generation processes with a shared set of parameters or constraints.

Built on

Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S · 2015
Earlier work this paper cites.
Denoising diffusion probabilistic models
Ho, J., Jain, A., and Abbeel, P · 2020
Earlier work this paper cites.
Denoising diffusion implicit models
Original
Song, J., Meng, C., and Ermon, S · 2020
Earlier work this paper cites.
Ilvr: Conditioning method for denoising diffusion probabilistic models
Original
Choi, J., Kim, S., Jeong, Y., Gwon, Y., and Yoon, S · 2021
Earlier work this paper cites.
Diffusion models beat gans on image synthesis
Dhariwal, P. and Nichol, A · 2021
Earlier work this paper cites.
Sdedit: Image synthesis and editing with stochastic differential equations
Original
Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S · 2021
Earlier work this paper cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M · 2021
Earlier work this paper cites.
Improved denoising diffusion probabilistic models
Nichol, A. Q. and Dhariwal, P · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Earlier work this paper cites.
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I · 2021
Earlier work this paper cites.
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., and Efros, A. A · 2022
Earlier work this paper cites.
Masked-attention mask transformer for universal image segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R · 2022
Earlier work this paper cites.
Diffedit: Diffusion-based semantic image editing with mask guidance
Original
Couairon, G., Verbeek, J., Schwenk, H., and Cord, M · 2022
Earlier work this paper cites.

Similar

Diffusion models in vision: A survey
Original
Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M · 2022
Cited alongside, same era.
Make-a-scene: Scene-based text-to-image generation with human priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., and Taigman, Y · 2022
Cited alongside, same era.
Prompt-to-prompt image editing with cross attention control
Original
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D · 2022
Cited alongside, same era.
Classifier-free diffusion guidance
Original
Ho, J. and Salimans, T · 2022
Cited alongside, same era.
Imagen video: High definition video generation with diffusion models
Original
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al · 2022
Cited alongside, same era.
Imagic: Text-based real image editing with diffusion models
Original
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M · 2022
Cited alongside, same era.
Diffusionclip: Text-guided diffusion models for robust image manipulation
Kim, G., Kwon, T., and Ye, J. C · 2022
Cited alongside, same era.
Diffusion-based image translation using disentangled style and content representation
Original
Kwon, G. and Ye, J. C · 2022
Cited alongside, same era.
Diffusion models already have a semantic latent space
Original
Kwon, M., Jeong, J., and Uh, Y · 2022
Cited alongside, same era.
Pseudo numerical methods for diffusion models on manifolds
Original
Liu, L., Ren, Y., Lin, Z., and Zhao, Z · 2022
Cited alongside, same era.
Null-text inversion for editing real images using guided diffusion models
Original
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D · 2022
Cited alongside, same era.
Diffrf: Rendering-guided 3d radiance field diffusion
Múller, N., , Siddiqui, Y., Porzi, L., Rota Buló, S., Kontschieder, P., and Nießner, M · 2022
Cited alongside, same era.

Then

Hierarchical text-conditional image generation with clip latents
Original
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B · 2022
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K · 2022
Later among the works it cites.
Palette: Image-to-image diffusion models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi, M · 2022
Later among the works it cites.
Laion-5b: An open large-scale dataset for training next generation image-text models
Original
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J · 2022
Later among the works it cites.
Knn-diffusion: Image generation via large-scale retrieval
Original
Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y · 2022
Later among the works it cites.
Make-a-video: Text-to-video generation without text-video data, 2022
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y · 2022
Later among the works it cites.
Human motion diffusion model
Original
Tevet, G., Raab, S., Gordon, B., Shafir, Y., Bermano, A. H., and Cohen-Or, D · 2022
Later among the works it cites.
Plug-and-play diffusion features for text-driven image-to-image translation
Original
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T · 2022
Later among the works it cites.
Unitune: Text-driven image editing by fine tuning an image generation model on a single image
Original
Valevski, D., Kalman, M., Matias, Y., and Leviathan, Y · 2022
Later among the works it cites.
Diffusers: State-of-the-art diffusion models
von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T · 2022
Later among the works it cites.
Physdiff: Physics-guided human motion diffusion model
Original
Yuan, Y., Song, J., Iqbal, U., Vahdat, A., and Kautz, J · 2022
Later among the works it cites.
Leveraging off-the-shelf diffusion model for multi-attribute fashion image manipulation
Kong, C., Jeon, D., Kwon, O., and Kwak, N · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…