2023

Delta Denoising Score

Hertz, Amir, Aberman, Kfir, Cohen-Or, Daniel

Understand

We introduce Delta Denoising Score (DDS), a novel scoring function for text-based image editing that guides minimal modifications of an input image towards the content described in a target prompt.

DDS leverages the rich generative prior of text-to-image diffusion models and can be used as a loss term in an optimization problem to steer an image towards a desired direction dictated by a text.
DDS utilizes the Score Distillation Sampling (SDS) mechanism for the purpose of image editing.
We show that using only SDS often produces non-detailed and blurry outputs due to noisy gradients.

Built on

Adaptive subgradient methods for online learning and stochastic optimization
John Duchi, Elad Hazan, and Yoram Singer · 2011
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Coco-stuff: Thing and stuff classes in context
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari · 2018
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Generative modeling by estimating gradients of the data distribution
Yang Song and Stefano Ermon · 2019
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
A method for stochastic optimization. arxiv 2014
Original
Diederik P Kingma, J Adam Ba, and J Adam · 2020
Earlier work this paper cites.
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Earlier work this paper cites.
Stylegan-nada: Clip-guided domain adaptation of image generators
Original
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or · 2021
Earlier work this paper cites.
Layered neural atlases for consistent video editing
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel · 2021
Earlier work this paper cites.
Clipstyler: Image style transfer with a single text condition
Original
Gihyun Kwon and Jong Chul Ye · 2021
Earlier work this paper cites.
Sdedit: Image synthesis and editing with stochastic differential equations
Original
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon · 2021
Earlier work this paper cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Earlier work this paper cites.

Similar

Styleclip: Text-driven manipulation of stylegan imagery
Original
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
High-resolution image synthesis with latent diffusion models, 2021
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2021
Cited alongside, same era.
Image super-resolution via iterative refinement
Original
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2021
Cited alongside, same era.
Blended latent diffusion
Original
Omri Avrahami, Ohad Fried, and Dani Lischinski · 2022
Cited alongside, same era.
Blended diffusion for text-driven editing of natural images
Omri Avrahami, Dani Lischinski, and Ohad Fried · 2022
Cited alongside, same era.
ediff-i: Text-to-image diffusion models with ensemble of expert denoisers
Original
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu · 2022
Cited alongside, same era.
Text2live: Text-driven layered image and video editing
Original
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel · 2022
Cited alongside, same era.
Instructpix2pix: Learning to follow image editing instructions
Original
Tim Brooks, Aleksander Holynski, and Alexei A Efros · 2022
Cited alongside, same era.
Diffedit: Diffusion-based semantic image editing with mask guidance
Original
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord · 2022
Cited alongside, same era.
Vqgan-clip: Open domain image generation and editing with natural language guidance
Original
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff · 2022
Cited alongside, same era.
Prompt-to-prompt image editing with cross attention control
Original
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Cited alongside, same era.

Then

Latent-nerf for shape-guided generation of 3d shapes and textures
Original
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or · 2022
Later among the works it cites.
Null-text inversion for editing real images using guided diffusion models
Original
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Later among the works it cites.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi · 2022
Later among the works it cites.
Diffusion guided domain adaptation of image generators
Original
Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, and Ahmed Elgammal · 2022
Later among the works it cites.
Motionclip: Exposing human motion generation to clip space
Original
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or · 2022
Later among the works it cites.
Plug-and-play diffusion features for text-driven image-to-image translation
Original
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel · 2022
Later among the works it cites.
Unsplash image collection, 2022
Unsplash · 2022
Later among the works it cites.
Unitune: Text-driven image editing by fine tuning an image generation model on a single image
Original
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan · 2022
Later among the works it cites.
Sketch-guided text-to-image diffusion models
Original
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or · 2022
Later among the works it cites.
Zero-shot image-to-image translation
Original
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu · 2023
Closest in time.
Dreambooth3d: Subject-driven text-to-3d generation
Original
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al · 2023
Closest in time.
p + p+ : Extended textual conditioning in text-to-image generation
Original
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…