2022

High-fidelity GAN Inversion with Padding Space

Bai, Qingyan, Xu, Yinghao, Zhu, Jiapeng et al.

Understand

Inverting a Generative Adversarial Network (GAN) facilitates a wide range of image editing tasks using pre-trained generators.

Existing methods typically employ the latent space of GANs as the inversion space yet observe the insufficient recovery of spatial details.
In this work, we propose to involve the padding space of the generator to complement the latent space with spatial information.
Concretely, we replace the constant padding (e.g., usually zeros) used in convolution layers with some instance-aware coefficients.

Built on

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Adv. Neural Inform. Process. Syst. (2014)
2014
Earlier work this paper cites.
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Int. Conf. Comput. Vis. (2015)
2015
Earlier work this paper cites.
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., Xiao, J.: LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)
Original
2015
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. (2016)
2016
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
Earlier work this paper cites.
Perarnau, G., Van De Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional GANs for image editing. In: Adv. Neural Inform. Process. Syst. Worksh. (2016)
2016
Earlier work this paper cites.
Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Eur. Conf. Comput. Vis. (2016)
2016
Earlier work this paper cites.
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Adv. Neural Inform. Process. Syst. (2017)
2017
Earlier work this paper cites.
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Int. Conf. Comput. Vis. pp. 1501–1510 (2017)
2017
Earlier work this paper cites.
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2117–2125 (2017)
2017
Earlier work this paper cites.
Lipton, Z.C., Tripathi, S.: Precise recovery of latent vectors from generative adversarial networks. In: Int. Conf. Learn. Represent. Worksh. (2017)
2017
Earlier work this paper cites.
Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. TNNLS (2018)
2018
Earlier work this paper cites.
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: Int. Conf. Learn. Represent. (2018)
2018
Earlier work this paper cites.
Roy, A.G., Navab, N., Wachinger, C.: Concurrent spatial and channel ‘squeeze & excitation’in fully convolutional networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 421–429. Springer (2018)
2018
Earlier work this paper cites.
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conf. Comput. Vis. Pattern Recog. (2018)
2018
Earlier work this paper cites.
Bau, D., Zhu, J.Y., Strobelt, H., Zhou, B., Tenenbaum, J.B., Freeman, W.T., Torralba, A.: GAN dissection: Visualizing and understanding generative adversarial networks. In: Int. Conf. Learn. Represent. (2019)
2019
Earlier work this paper cites.
Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: Int. Conf. Learn. Represent. (2019)
2019
Earlier work this paper cites.
Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. (2019)
2019
Earlier work this paper cites.
Donahue, J., Simonyan, K.: Large scale adversarial representation learning. In: Adv. Neural Inform. Process. Syst. (2019)
2019
Earlier work this paper cites.
Islam, M.A., Jia, S., Bruce, N.D.: How much position information do convolutional neural networks encode? Int. Conf. Learn. Represent. (2019)
2019
Earlier work this paper cites.

Similar

Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE Conf. Comput. Vis. Pattern Recog. (2019)
2019
Cited alongside, same era.
Rameen, A., Yipeng, Q., Peter, W.: Image2StyleGAN: How to embed images into the stylegan latent space? In: Int. Conf. Comput. Vis. (2019)
2019
Cited alongside, same era.
Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN++: How to edit the embedded images? In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
2020
Cited alongside, same era.
Gu, J., Shen, Y., Zhou, B.: Image processing using multi-code GAN prior. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
2020
Cited alongside, same era.
Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: GANSpace: Discovering interpretable GAN controls. In: Adv. Neural Inform. Process. Syst. (2020)
2020
Cited alongside, same era.
Huh, M., Zhang, R., Zhu, J.Y., Paris, S., Hertzmann, A.: Transforming and projecting images to class-conditional generative networks. In: Eur. Conf. Comput. Vis. (2020)
2020
Cited alongside, same era.
Jahanian, A., Chai, L., Isola, P.: On the "steerability" of generative adversarial networks. In: Int. Conf. Learn. Represent. (2020)
2020
Cited alongside, same era.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
2020
Cited alongside, same era.
Kayhan, O.S., Gemert, J.C.v.: On translation invariance in CNNs: Convolutional layers can exploit absolute spatial location. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
2020
Cited alongside, same era.
Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. In: Eur. Conf. Comput. Vis. (2020)
2020
Cited alongside, same era.
Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. In: Eur. Conf. Comput. Vis. (2020)
2020
Cited alongside, same era.
Pidhorskyi, S., Adjeroh, D.A., Doretto, G.: Adversarial latent autoencoders. In: IEEE Conf. Comput. Vis. Pattern Recog. (2020)
2020
Cited alongside, same era.

Then

Kang, K., Kim, S., Cho, S.: GAN inversion for out-of-range images with geometric transformations. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 13941–13949 (2021)
2021
Later among the works it cites.
Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. In: Adv. Neural Inform. Process. Syst. (2021)
2021
Later among the works it cites.
Ling, H., Kreis, K., Li, D., Kim, S.W., Torralba, A., Fidler, S.: EditGAN: High-precision semantic image editing. In: Adv. Neural Inform. Process. Syst. (2021)
2021
Later among the works it cites.
Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
2021
Later among the works it cites.
Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744 (2021)
Original
2021
Later among the works it cites.
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph. (2021)
2021
Later among the works it cites.
Wu, Z., Lischinski, D., Shechtman, E.: StyleSpace analysis: Disentangled controls for StyleGAN image generation. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
2021
Later among the works it cites.
Xu, R., Wang, X., Chen, K., Zhou, B., Loy, C.C.: Positional encoding as spatial inductive bias in GANs. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
2021
Later among the works it cites.
Xu, Y., Shen, Y., Zhu, J., Yang, C., Zhou, B.: Generative hierarchical features from synthesizing images. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
2021
Later among the works it cites.
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.F., Barriuso, A., Torralba, A., Fidler, S.: DatasetGAN: Efficient labeled data factory with minimal human effort. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
2021
Later among the works it cites.
Zhu, J., Feng, R., Shen, Y., Zhao, D., Zha, Z., Zhou, J., Chen, Q.: Low-rank subspaces in GANs. In: Adv. Neural Inform. Process. Syst. (2021)
2021
Later among the works it cites.
Zhu, P., Abdal, R., Femiani, J., Wonka, P.: Barbershop: GAN-based image compositing using segmentation masks. arXiv preprint arXiv:2106.01505 (2021)
Original
2021
Later among the works it cites.
Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.: Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 18511–18521 (2022)
2022
Closest in time.
Dinh, T.M., Tran, A.T., Nguyen, R., Hua, B.S.: Hyperinverter: Improving stylegan inversion via hypernetwork. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 11389–11398 (2022)
2022
Closest in time.
Shen, Y., Zhang, Z., Yang, D., Xu, Y., Yang, C., Zhu, J.: Hammer: An efficient toolkit for training deep models. https://github.com/bytedance/Hammer (2022)
2022
Closest in time.
Wang, T., Zhang, Y., Fan, Y., Wang, J., Chen, Q.: High-fidelity GAN inversion for image attribute editing. In: IEEE Conf. Comput. Vis. Pattern Recog. (2022)
2022
Closest in time.
Xia, W., Zhang, Y., Yang, Y., Xue, J.H., Zhou, B., Yang, M.H.: GAN Inversion: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (2022)
2022
Closest in time.
Zhu, J., Shen, Y., Xu, Y., Zhao, D., Chen, Q.: Region-based semantic factorization in GANs. In: Int. Conf. Mach. Learn. (2022)
2022
Closest in time.
Zhu, J., Zhao, D., Zhang, B., Zhou, B.: Disentangled inference for GANs with latently invertible autoencoder. Int. J. Comput. Vis. (2022)
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…