2021

Image-Based CLIP-Guided Essence Transfer

Chefer, Hila, Benaim, Sagie, Paiss, Roni et al.

Understand

We make the distinction between (i) style transfer, in which a source image is manipulated to match the textures and colors of a target image, and (ii) essence transfer, in which one edits the source image to include high-level semantic attributes from the target.

Crucially, the semantic attributes that constitute the essence of an image may differ from image to image.
Our blending operator combines the powerful StyleGAN generator and the semantic encoder of CLIP in a novel way that is simultaneously additive in both latent spaces, resulting in a mechanism that guarantees both identity preservation and high-level feature transfer without relying on a facial recognition network.
We present two variants of our method.

Built on

Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece. vol. 2, pp. 1033–1038 vol.2 (1999). https://doi.org/10.1109/ICCV.1999.790383
1999
Earlier work this paper cites.
Ullman, S.: High-level vision: Object recognition and visual cognition. MIT press (2000)
2000
Earlier work this paper cites.
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing Machinery, New York, NY, USA (2001). https://doi.org/10.1145/383259.383295, https://doi.org/10.1145/383259.383295
2001
Earlier work this paper cites.
Sunkavalli, K., Johnson, M.K., Matusik, W., Pfister, H.: Multi-scale image harmonization. ACM Trans. Graph. 29
2010
Earlier work this paper cites.
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980
Original
2015
Earlier work this paper cites.
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)
2015
Earlier work this paper cites.
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (6 2016)
2016
Earlier work this paper cites.
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)
2016
Earlier work this paper cites.
Perarnau, G., van de Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional gans for image editing. ArXiv abs/1611.06355
Original
2016
Earlier work this paper cites.
Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: ECCV (2016)
2016
Earlier work this paper cites.
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., Krishnan, D.: Unsupervised pixel-level domain adaptation with generative adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 95–104 (2017)
2017
Earlier work this paper cites.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 6629–6640. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
2017
Earlier work this paper cites.
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy. pp. 1510–1519 (2017). https://doi.org/10.1109/ICCV.2017.167
2017
Earlier work this paper cites.
Huang, X., Belongie, S.J.: Arbitrary style transfer in real-time with adaptive instance normalization. 2017 IEEE International Conference on Computer Vision (ICCV) pp. 1510–1519 (2017)
2017
Earlier work this paper cites.
Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Universal style transfer via feature transforms. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 385–395. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
2017
Earlier work this paper cites.
Lipton, Z.C., Tripathi, S.: Precise recovery of latent vectors from generative adversarial networks. ArXiv abs/1702.04782
Original
2017
Earlier work this paper cites.
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA. pp. 6997–7005 (2017). https://doi.org/10.1109/CVPR.2017.740
2017
Earlier work this paper cites.
Luo, J., Xu, Y., Tang, C., Lv, J.: Learning inverse mapping by autoencoder based generative adversarial nets. In: International Conference on Neural Information Processing. pp. 207–216. Springer (2017)
2017
Earlier work this paper cites.

Similar

Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep painterly harmonization. Computer Graphics Forum 37
2018
Cited alongside, same era.
Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: How to embed images into the stylegan latent space? 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 4431–4440 (2019)
2019
Cited alongside, same era.
Bau, D., Strobelt, H., Peebles, W.S., Wulff, J., Zhou, B., Zhu, J.Y., Torralba, A.: Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG) 38
2019
Cited alongside, same era.
Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems 30
2019
Cited alongside, same era.
Deng, J., Guo, J., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4685–4694 (2019)
2019
Cited alongside, same era.
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4396–4405 (2019)
2019
Cited alongside, same era.
Collins, E., Bala, R., Price, B., Susstrunk, S.: Editing in style: Uncovering the local semantics of gans. In: CVPR. pp. 5771–5780 (2020)
2020
Cited alongside, same era.
Gu, J., Shen, Y., Zhou, B.: Image processing using multi-code gan prior. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3009–3018 (2020)
2020
Cited alongside, same era.
Guan, S., Tai, Y., Ni, B., Zhu, F., Huang, F., Yang, X.: Collaborative learning for faster stylegan embedding. ArXiv abs/2007.01758
Original
2020
Cited alongside, same era.
Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: Ganspace: Discovering interpretable gan controls. arXiv preprint arXiv:2004.02546 (2020)
Original
2020
Cited alongside, same era.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: CVPR. pp. 8110–8119 (2020)
2020
Cited alongside, same era.
Kim, S.S.Y., Kolkin, N., Salavon, J., Shakhnarovich, G.: Deformable style transfer. In: European Conference on Computer Vision (ECCV) (2020)
2020
Cited alongside, same era.

Then

Alaluf, Y., Patashnik, O., Cohen-Or, D.: Restyle: A residual-based stylegan encoder via iterative refinement. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 6691–6700 (2021)
2021
Closest in time.
Chong, M.J., Forsyth, D.: Jojogan: One shot face stylization. ArXiv 2112.11641
2021
Closest in time.
Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators (2021)
2021
Closest in time.
Kang, K., Kim, S., Cho, S.: Gan inversion for out-of-range images with geometric transformations. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 13921–13929 (2021)
2021
Closest in time.
Kim, H., Choi, Y., Kim, J., Yoo, S., Uh, Y.: Exploiting spatial dimensions of latent in gan for real-time image editing. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 852–861 (2021)
2021
Closest in time.
Liu, M., Li, Q., Qin, Z., Zhang, G., Wan, P., Zheng, W.: Blendgan: Implicitly gan blending for arbitrary stylized face generation. In: Advances in Neural Information Processing Systems (2021)
2021
Closest in time.
Ojha, U., Li, Y., Lu, J., Efros, A.A., Lee, Y.J., Shechtman, E., Zhang, R.: Few-shot image generation via cross-domain correspondence. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 10738–10747 (2021)
2021
Closest in time.
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., Lischinski, D.: Styleclip: Text-driven manipulation of stylegan imagery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2085–2094 (2021)
2021
Closest in time.
Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)
Original
2021
Closest in time.
Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a stylegan encoder for image-to-image translation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2021)
2021
Closest in time.
Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744 (2021)
Original
2021
Closest in time.
Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in gans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1532–1540 (2021)
2021
Closest in time.
Tewel, Y., Shalev, Y., Schwartz, I., Wolf, L.: Zero-shot image-to-text generation for visual-semantic arithmetic. In: CVPR (2021)
2021
Closest in time.
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766 (2021)
Original
2021
Closest in time.
Wang, T., Zhang, Y., Fan, Y., Wang, J., Chen, Q.: High-fidelity gan inversion for image attribute editing. ArXiv abs/2109.06590
Original
2021
Closest in time.
Zhu, P., Abdal, R., Femiani, J.C., Wonka, P.: Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. ArXiv 2110.08398
2021
Closest in time.
Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation (2022)
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…