2023

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Gal, Rinon, Arar, Moab, Atzmon, Yuval et al.

Understand

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts.

However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity.
To overcome these limitations, we propose an encoder-based domain-tuning approach.
Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain.

Built on

Image Processing Using Multi-Code GAN Prior
Original
Jinjin Gu, Yujun Shen, and Bolei Zhou. 2020 · 1912
Earlier work this paper cites.
A personalized recommender system based on web usage mining and decision tree induction
Yoon Ho Cho, Jae Kyeong Kim, and Soung Hie Kim. 2002 · 2002
Earlier work this paper cites.
Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real-Life’Images: detection, alignment, and recognition
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008 · 2008
Earlier work this paper cites.
What’s on TV tonight? An efficient and effective personalized recommender system of TV programs
Ana Belen Barragans Martinez, Jose J Pazos Arias, Ana Fernandez Vilas, Jorge Garcia Duque, and Martin Lopez Nores. 2009 · 2009
Earlier work this paper cites.
Improved StyleGAN Embedding: Where are the Good Latents?
Original
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020a · 2012
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Large-scale classification of fine-art paintings: Learning the right metric on the right feature
Original
Babak Saleh and Ahmed Elgammal. 2015 · 2015
Earlier work this paper cites.
Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
Original
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. 2015 · 2015
Earlier work this paper cites.
Hypernetworks
Original
David Ha, Andrew Dai, and Quoc V Le. 2016 · 2016
Earlier work this paper cites.
Generative visual manipulation on the natural image manifold. In European conference on computer vision . Springer, 597–613
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. 2016 · 2016
Earlier work this paper cites.
Personalized recommender system for e-Learning environment
Soulef Benhamdi, Abdesselam Babouri, and Raja Chiky. 2017 · 2017
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning . PMLR, 1126–1135
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017 · 2017
Earlier work this paper cites.
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 6626–6637
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017 · 2017
Earlier work this paper cites.
Progressive growing of gans for improved quality, stability, and variation
Original
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017 · 2017
Earlier work this paper cites.
Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 6306–6315
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017 · 2017
Earlier work this paper cites.
Artwork Personalization at Netflix. In Proceedings of the 12th ACM Conference on Recommender Systems (Vancouver, British Columbia, Canada) (RecSys ’18) . Association for Computing Machinery, New York, NY, USA, 487–488
Fernando Amat, Ashok Chandrashekar, Tony Jebara, and Justin Basilico. 2018 · 2018
Earlier work this paper cites.
Reptile: a scalable metalearning algorithm
Original
Alex Nichol and John Schulman. 2018 · 2018
Earlier work this paper cites.
Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition . 9446–9454
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018 · 2018
Earlier work this paper cites.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1316–1324
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018 · 2018
Earlier work this paper cites.
Image2stylegan: How to embed images into the stylegan latent space?. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 4432–4441
Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019 · 2019
Earlier work this paper cites.
Semantic Photo Manipulation with a Generative Image Prior
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2019 · 2019
Earlier work this paper cites.
Large Scale GAN Training for High Fidelity Natural Image Synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019 · 2019
Earlier work this paper cites.
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Original
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019 · 2019
Earlier work this paper cites.
Improving federated learning personalization via model agnostic meta learning
Original
Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan. 2019 · 2019
Earlier work this paper cites.
A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4401–4410
Tero Karras, Samuli Laine, and Timo Aila. 2019 · 2019
Earlier work this paper cites.
On the spectral bias of neural networks. In International Conference on Machine Learning . PMLR, 5301–5310
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. 2019 · 2019
Earlier work this paper cites.
Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 5802–5810
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. 2019 · 2019
Earlier work this paper cites.

Similar

Image2stylegan++: How to edit the embedded images?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 8296–8305
Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020 · 2020
Cited alongside, same era.
Personalized federated learning: A meta-learning approach
Original
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020 · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020 · 2020
Cited alongside, same era.
Curricularface: adaptive curriculum learning loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 5901–5910
Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. 2020 · 2020
Cited alongside, same era.
Three approaches for personalization with applications to federated learning
Original
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. 2020 · 2020
Cited alongside, same era.
Adversarial Latent Autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14104–14113
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. 2020 · 2020
Cited alongside, same era.
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation
Original
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2020 · 2020
Cited alongside, same era.
Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9243–9252
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. 2020 · 2020
Cited alongside, same era.
Denoising Diffusion Implicit Models. In International Conference on Learning Representations
Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020 · 2020
Cited alongside, same era.
Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis
Original
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao. 2020 · 2020
Cited alongside, same era.
In-domain gan inversion for real image editing
Original
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020b · 2020
Cited alongside, same era.
Known unknowns: Learning novel concepts using reasoning-by-elimination. In Conference on Uncertainty in Artificial Intelligence
Harsh Agrawal, Eli A. Meirom, Yuval Atzmon, Shie Mannor, and Gal Chechik. 2021 · 2021
Cited alongside, same era.

Then

Designing an Encoder for StyleGAN Image Manipulation
Original
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021 · 2021
Later among the works it cites.
GAN Inversion: A Survey
Original
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. 2021 · 2021
Later among the works it cites.
Generative Hierarchical Features from Synthesizing Images. In CVPR
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. 2021 · 2021
Later among the works it cites.
Improving text-to-image synthesis using contrastive learning
Original
Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. 2021 · 2021
Later among the works it cites.
Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 833–842
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. 2021 · 2021
Later among the works it cites.
State-of-the-Art in the Architecture, Methods and Applications of StyleGAN. In Computer Graphics Forum , Vol. 41. Wiley Online Library, 591–611
2022 · 2022
Later among the works it cites.
ediffi: Text-to-image diffusion models with an ensemble of expert denoisers
Original
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al · 2022
Later among the works it cites.
Authentic Volumetric Avatars From a Phone Scan
Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shunsuke Saito, Stephen Lombardi, Shih-en Wei, Danielle Belko, Shoou-i Yu, Yaser Sheikh, and Jason Saragih. 2022 · 2022
Later among the works it cites.
"This is my unicorn, Fluffy": Personalizing frozen vision-language representations. In European Conference on Computer Vision (ECCV)
Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. 2022 · 2022
Later among the works it cites.
Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11389–11398
Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. 2022 · 2022
Later among the works it cites.
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV (Lecture Notes in Computer Science, Vol. 13675) , Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 89–106
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. 2022 · 2022
Later among the works it cites.
An image is worth one word: Personalizing text-to-image generation using textual inversion
Original
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
Multi-Concept Customization of Text-to-Image Diffusion
Original
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2022 · 2022
Later among the works it cites.
Null-text Inversion for Editing Real Images using Guided Diffusion Models
Original
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
MyStyle: A Personalized Generative Prior
Original
Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. 2022 · 2022
Later among the works it cites.
Spatially-Adaptive Multilayer Selection for GAN Inversion and Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11399–11409
Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, and Krishna Kumar Singh. 2022 · 2022
Later among the works it cites.
HuggingFace concept library
Apolinário Passos and Omar Sanseviero. 2022 · 2022
Later among the works it cites.
HuggingFace DreamBooth Implementation
Suraj Patil and Pedro Cuenca. 2022 · 2022
Later among the works it cites.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022 · 2022
Later among the works it cites.
High-Resolution Image Synthesis with Latent Diffusion Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 . IEEE, 10674–10685
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022 · 2022
Later among the works it cites.
DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022 · 2022
Later among the works it cites.
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Original
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al · 2022
Later among the works it cites.
Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10748–10757
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2022 · 2022
Later among the works it cites.
CLIPascene: Scene Sketching with Different Types and Levels of Abstraction
Original
Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. 2022 · 2022
Later among the works it cites.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Original
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al · 2022
Later among the works it cites.
GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models
Original
Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…