2022

Paint by Example: Exemplar-based Image Editing with Diffusion Models

Yang, Binxin, Gu, Shuyang, Zhang, Bo et al.

Understand

Language-guided image editing has achieved great success recently.

In this paper, for the first time, we investigate exemplar-guided image editing for more precise control.
We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar.
However, the naive approach will cause obvious fusing artifacts.

Built on

Color transfer between images
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley · 2001
Earlier work this paper cites.
Poisson image editing
Patrick Pérez, Michel Gangnet, and Andrew Blake · 2003
Earlier work this paper cites.
Color harmonization
Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, and Ying-Qing Xu · 2006
Earlier work this paper cites.
Drag-and-drop pasting
Jiaya Jia, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum · 2006
Earlier work this paper cites.
Multi-scale image harmonization
Kalyan Sunkavalli, Micah K Johnson, Wojciech Matusik, and Hanspeter Pfister · 2010
Earlier work this paper cites.
Error-tolerant image compositing
Michael W Tao, Micah K Johnson, and Sylvain Paris · 2010
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Colorful image colorization
Richard Zhang, Phillip Isola, and Alexei A Efros · 2016
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Deep image harmonization
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang · 2017
Earlier work this paper cites.
Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans
Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang · 2018
Earlier work this paper cites.
Aesthetic-driven image enhancement by adversarial learning
Yubin Deng, Chen Change Loy, and Xiaoou Tang · 2018
Earlier work this paper cites.
Faceid-gan: Learning a symmetry three-player gan for identity-preserving face synthesis
Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang, and Xiaoou Tang · 2018
Earlier work this paper cites.
Generative image inpainting with contextual attention
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang · 2018
Earlier work this paper cites.
Toward realistic image compositing with adversarial learning
Bor-Chun Chen and Andrew Kae · 2019
Earlier work this paper cites.
Mask-guided portrait editing with conditional gans
Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan · 2019
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Faceshifter: Towards high fidelity and occlusion aware face swapping
Original
Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen · 2019
Earlier work this paper cites.
Deep exemplar-based video colorization
Bo Zhang, Mingming He, Jing Liao, Pedro V Sander, Lu Yuan, Amine Bermak, and Dong Chen · 2019
Earlier work this paper cites.
Semantic photo manipulation with a generative image prior
Original
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba · 2020
Earlier work this paper cites.
Albumentations: Fast and flexible image augmentations
Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin · 2020
Earlier work this paper cites.
Dovenet: Deep image harmonization via domain verification
Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang · 2020
Earlier work this paper cites.
Improving the harmony of the composite image by spatial-separated attention module
Xiaodong Cun and Chi-Man Pun · 2020
Earlier work this paper cites.
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2020
Earlier work this paper cites.

Similar

Giqa: Generated image quality assessment
Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Analyzing and improving the image quality of stylegan
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila · 2020
Cited alongside, same era.
The open images dataset v4
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al · 2020
Cited alongside, same era.
Interpreting the latent space of gans for semantic face editing
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou · 2020
Cited alongside, same era.
Score-based generative modeling through stochastic differential equations
Original
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2020
Cited alongside, same era.
Cross-domain correspondence learning for exemplar-based image translation
Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen · 2020
Cited alongside, same era.
Paint by word
Original
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba · 2021
Cited alongside, same era.
High-fidelity and arbitrary face editing
Yue Gao, Fangyun Wei, Jianmin Bao, Shuyang Gu, Dong Chen, Fang Wen, and Zhouhui Lian · 2021
Cited alongside, same era.
Image harmonization with transformer
Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, and Junyu Dong · 2021
Cited alongside, same era.
Intrinsic image harmonization
Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng · 2021
Cited alongside, same era.
Editgan: High-precision semantic image editing
Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler · 2021
Cited alongside, same era.

Then

Cocosnet v2: Full-resolution correspondence learning for image translation
Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen · 2021
Later among the works it cites.
Clip2stylegan: Unsupervised extraction of stylegan edit directions
Rameen Abdal, Peihao Zhu, John Femiani, Niloy Mitra, and Peter Wonka · 2022
Closest in time.
Hyperstyle: Stylegan inversion with hypernetworks for real image editing
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano · 2022
Closest in time.
Blended diffusion for text-driven editing of natural images
Omri Avrahami, Dani Lischinski, and Ohad Fried · 2022
Closest in time.
Make-a-scene: Scene-based text-to-image generation with human priors
Original
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman · 2022
Closest in time.
Stylegan-nada: Clip-guided domain adaptation of image generators
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or · 2022
Closest in time.
Vector quantized diffusion model for text-to-image synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo · 2022
Closest in time.
Prompt-to-prompt image editing with cross attention control
Original
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Closest in time.
Classifier-free diffusion guidance
Original
Jonathan Ho and Tim Salimans · 2022
Closest in time.
Guidedstyle: Attribute knowledge guided style manipulation for semantic face editing
Xianxu Hou, Xiaokang Zhang, Hanbang Liang, Linlin Shen, Zhihui Lai, and Jun Wan · 2022
Closest in time.
Imagic: Text-based real image editing with diffusion models
Original
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani · 2022
Closest in time.
Diffusionclip: Text-guided diffusion models for robust image manipulation
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye · 2022
Closest in time.
The role of imagenet classes in fr
Original
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
Pivotal tuning for latent-based editing of real images
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or · 2022
Closest in time.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Closest in time.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Closest in time.
Palette: Image-to-image diffusion models
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi · 2022
Closest in time.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Closest in time.
Resolution-robust large mask inpainting with fourier convolutions
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky · 2022
Closest in time.
Improved vector quantized diffusion models
Original
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen · 2022
Closest in time.
Pretraining is all you need for image-to-image translation
Original
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen · 2022
Closest in time.
Dccf: Deep comprehensible color filter learning framework for high-resolution image harmonization
Original
Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, and Xing Tang · 2022
Closest in time.
Scaling autoregressive models for content-rich text-to-image generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al · 2022
Closest in time.
Styleswin: Transformer-based gan for high-resolution image generation
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…