2021

Blended Diffusion for Text-driven Editing of Natural Images

Avrahami, Omri, Lischinski, Dani, Fried, Ohad

Understand

Natural language offers a highly intuitive interface for image editing.

In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask.
We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results.
To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels.

Built on

On the theory of stochastic processes, with particular reference to applications
W Feller · 1949
Earlier work this paper cites.
Comparing individual means in the analysis of variance
John W Tukey · 1949
Earlier work this paper cites.
The Laplacian pyramid as a compact image code
Peter J Burt and Edward H Adelson · 1987
Earlier work this paper cites.
Poisson image editing
Patrick Pérez, Michel Gangnet, and Andrew Blake · 2003
Earlier work this paper cites.
Scene completion using millions of photographs
James Hays and Alexei A. Efros · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Coordinates for instant image cloning
Zeev Farbman, Gil Hoffer, Yaron Lipman, Daniel Cohen-Or, and Dani Lischinski · 2009
Earlier work this paper cites.
Digital doctoring: can we trust photographs?
Hany Farid · 2009
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy · 2014
Earlier work this paper cites.
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus · 2014
Earlier work this paper cites.
Learning deep features for scene recognition using places database
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Generating images from captions with attention
Original
Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov · 2016
Earlier work this paper cites.
Generative adversarial text to image synthesis
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee · 2016
Earlier work this paper cites.
GANs trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Neural discrete representation learning
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2017
Earlier work this paper cites.
Large scale GAN training for high fidelity natural image synthesis
Original
Andrew Brock, Jeff Donahue, and Karen Simonyan · 2018
Earlier work this paper cites.

Similar

AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He · 2018
Cited alongside, same era.
StackGAN++: Realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas · 2018
Cited alongside, same era.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Cited alongside, same era.
Photographic text-to-image synthesis with a hierarchically-nested adversarial network
Zizhao Zhang, Yuanpu Xie, and Lin Yang · 2018
Cited alongside, same era.
Image2stylegan: How to embed images into the StyleGAN latent space?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2019
Cited alongside, same era.
Semantic object accuracy for generative text-to-image synthesis
Original
Tobias Hinz, Stefan Heinrich, and Stefan Wermter · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Cited alongside, same era.
Controllable text-to-image generation
Original
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr · 2019
Cited alongside, same era.
Object-driven text-to-image synthesis via adversarial training
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao · 2019
Cited alongside, same era.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Cited alongside, same era.
Learn, imagine and create: Text-to-image generation from prior knowledge
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao · 2019
Cited alongside, same era.
MirrorGAN: Learning text-to-image generation by redescription
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao · 2019
Cited alongside, same era.

Then

In-domain GAN inversion for real image editing
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou · 2020
Later among the works it cites.
Restyle: A residual-based StyleGAN encoder via iterative refinement
Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or · 2021
Closest in time.
Paint by word
Original
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba · 2021
Closest in time.
Trump is reportedly suggesting the ‘access hollywood’ tape was fake news
Aaron Blake · 2021
Closest in time.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Closest in time.
Cogview: Mastering text-to-image generation via transformers
Original
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al · 2021
Closest in time.
Taming transformers for high-resolution image synthesis
Patrick Esser, Robin Rombach, and Bjorn Ommer · 2021
Closest in time.
Multimodal neurons in artificial neural networks
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah · 2021
Closest in time.
Alias-free generative adversarial networks
Original
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila · 2021
Closest in time.
Diffusionclip: Text-guided image manipulation using diffusion models
Original
Gwanghyun Kim and Jong Chul Ye · 2021
Closest in time.
More control for free! image synthesis with semantic diffusion guidance
Original
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell · 2021
Closest in time.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Closest in time.
Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal · 2021
Closest in time.
StyleCLIP: Text-driven manipulation of StyleGAN imagery
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski · 2021
Closest in time.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Closest in time.
Zero-shot text-to-image generation
Original
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Closest in time.
Encoding in style: a StyleGAN encoder for image-to-image translation
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or · 2021
Closest in time.
Designing an encoder for StyleGAN image manipulation
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or · 2021
Closest in time.
Towards open-world text-guided face image generation and manipulation
Original
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu · 2021
Closest in time.
GAN inversion: A survey
Original
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…