2023

P+: Extended Textual Conditioning in Text-to-Image Generation

Voynov, Andrey, Chu, Qinghao, Cohen-Or, Daniel et al.

Understand

We introduce an Extended Textual Conditioning space in text-to-image models, referred to as $P+$.

This space consists of multiple textual conditions, derived from per-layer prompts, each corresponding to a layer of the denoising U-net of the diffusion model.
We show that the extended space provides greater disentangling and control over image synthesis.
We further introduce Extended Textual Inversion (XTI), where the images are inverted into $P+$, and represented by per-layer tokens.

Built on

Remarks on some nonparametric estimates of a density function
Murray Rosenblatt · 1956
Earlier work this paper cites.
On estimation of a probability density function and mode
Emanuel Parzen · 1962
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Generative visual manipulation on the natural image manifold
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros · 2016
Earlier work this paper cites.
Precise recovery of latent vectors from generative adversarial networks
Original
Zachary C Lipton and Subarna Tripathi · 2017
Earlier work this paper cites.
Semantic image inpainting with deep generative models, 2017
Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, and Minh N. Do · 2017
Earlier work this paper cites.
Inverting the generator of a generative adversarial network
Antonia Creswell and Anil Anthony Bharath · 2018
Earlier work this paper cites.
Image2stylegan: How to embed images into the stylegan latent space?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2019
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Earlier work this paper cites.
Generative modeling by estimating gradients of the data distribution
Yang Song and Stefano Ermon · 2019
Earlier work this paper cites.
Image2stylegan++: How to edit the embedded images?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2020
Earlier work this paper cites.
Understanding the role of individual units in a deep neural network
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
Training generative adversarial networks with limited data
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila · 2020
Earlier work this paper cites.
Encoding in style: a stylegan encoder for image-to-image translation
Original
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or · 2020
Earlier work this paper cites.
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2020
Earlier work this paper cites.
Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano · 2021
Earlier work this paper cites.
Emerging properties in self-supervised vision transformers
Original
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin · 2021
Earlier work this paper cites.
Classifier-free diffusion guidance
Jonathan Ho and Tim Salimans · 2021
Earlier work this paper cites.

Similar

Lora: Low-rank adaptation of large language models, 2021
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2021
Cited alongside, same era.
Diffusionclip: Text-guided image manipulation using diffusion models
Original
Gwanghyun Kim and Jong Chul Ye · 2021
Cited alongside, same era.
Clipstyler: Image style transfer with a single text condition
Original
Gihyun Kwon and Jong Chul Ye · 2021
Cited alongside, same era.
Sdedit: Image synthesis and editing with stochastic differential equations
Original
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon · 2021
Cited alongside, same era.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Cited alongside, same era.
Styleclip: Text-driven manipulation of stylegan imagery
Original
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Original
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Cited alongside, same era.
High-resolution image synthesis with latent diffusion models, 2021
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2021
Cited alongside, same era.
Designing an encoder for stylegan image manipulation
Original
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or · 2021
Cited alongside, same era.
High-fidelity gan inversion for image attribute editing
Original
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen · 2021
Cited alongside, same era.
Stylespace analysis: Disentangled controls for stylegan image generation
Zongze Wu, Dani Lischinski, and Eli Shechtman · 2021
Cited alongside, same era.
Gan inversion: A survey, 2021
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang · 2021
Cited alongside, same era.

Then

Imagic: Text-based real image editing with diffusion models
Original
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani · 2022
Later among the works it cites.
Imagic: Text-based real image editing with diffusion models
Original
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui-Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Irani · 2022
Later among the works it cites.
Magic3d: High-resolution text-to-3d content creation
Original
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin · 2022
Later among the works it cites.
Pseudo numerical methods for diffusion models on manifolds, 2022
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao · 2022
Later among the works it cites.
Null-text inversion for editing real images using guided diffusion models
Original
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or · 2022
Later among the works it cites.
Mystyle: A personalized generative prior
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Later among the works it cites.
Pivotal tuning for latent-based editing of real images
Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or · 2022
Later among the works it cites.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Later among the works it cites.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Original
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Later among the works it cites.
Low-rank adaptation for fast text-to-image diffusion fine-tuning
Simo Ryu · 2022
Later among the works it cites.
Photorealistic text-to-image diffusion models with deep language understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi · 2022
Later among the works it cites.
Plug-and-play diffusion features for text-driven image-to-image translation
Original
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel · 2022
Later among the works it cites.
Unitune: Text-driven image editing by fine tuning an image generation model on a single image
Original
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan · 2022
Later among the works it cites.
Imagen editor and editbench: Advancing and evaluating text-guided image inpainting
Original
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al · 2022
Later among the works it cites.
Instructpix2pix: Learning to follow image editing instructions
Tim Brooks, Aleksander Holynski, and Alexei A. Efros · 2023
Closest in time.
Muse: Text-to-image generation via masked generative transformers
Original
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al · 2023
Closest in time.
Designing an encoder for fast personalization of text-to-image models
Original
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or · 2023
Closest in time.
Multi-concept customization of text-to-image diffusion
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu · 2023
Closest in time.
Zero-shot image-to-image translation
Original
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…