2021

EdiBERT, a generative model for image editing

Issenhuth, Thibaut, Tanielian, Ugo, Mary, Jérémie et al.

Understand

Advances in computer vision are pushing the limits of im-age manipulation, with generative models sampling detailed images on various tasks.

However, a specialized model is often developed and trained for each specific task, even though many image edition tasks share similarities.
In denoising, inpainting, or image compositing, one always aims at generating a realistic image from a low-quality one.
In this paper, we aim at making a step towards a unified approach for image editing.

Built on

The laplacian pyramid as a compact image code
Peter J Burt and Edward H Adelson · 1987
Earlier work this paper cites.
A non-local algorithm for image denoising
Antoni Buades, Bartomeu Coll, and J-M Morel · 2005
Earlier work this paper cites.
Scene completion using millions of photographs
James Hays and Alexei A Efros · 2007
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
Original
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao · 2015
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter · 2017
Earlier work this paper cites.
Neural discrete representation learning
Aaron Van den Oord, Oriol Vinyals, and Koray Kavukcuoglu · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros · 2017
Earlier work this paper cites.
Image transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran · 2018
Earlier work this paper cites.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
Image2stylegan: How to embed images into the stylegan latent space?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2019
Earlier work this paper cites.

Similar

Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Cited alongside, same era.
On the" steerability" of generative adversarial networks
Ali Jahanian, Lucy Chai, and Phillip Isola · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
T. Karras, S. Laine, and T. Aila · 2019
Cited alongside, same era.
Bert has a mouth, and it must speak: Bert as a markov random field language model
Original
Alex Wang and Kyunghyun Cho · 2019
Cited alongside, same era.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le · 2019
Cited alongside, same era.
Self-attention generative adversarial networks
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena · 2019
Cited alongside, same era.
Image2stylegan++: How to edit the embedded images?
Rameen Abdal, Yipeng Qin, and Peter Wonka · 2020
Cited alongside, same era.
The epistemic threat of deepfakes
Don Fallis · 2020
Cited alongside, same era.
Reliable fidelity and diversity metrics for generative models
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo · 2020
Cited alongside, same era.
Interpreting the latent space of gans for semantic face editing
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou · 2020
Cited alongside, same era.
Large scale image completion via co-modulated generative adversarial networks
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, I Eric, Chao Chang, and Yan Xu · 2020
Cited alongside, same era.
In-domain gan inversion for real image editing
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou · 2020
Cited alongside, same era.

Then

Using latent space regression to analyze and leverage compositionality in gans
Original
Lucy Chai, Jonas Wulff, and Phillip Isola · 2021
Closest in time.
Intermediate layer optimization for inverse problems using deep generative models
Giannis Daras, Joseph Dean, Ajil Jalal, and Alex Dimakis · 2021
Closest in time.
Generative adversarial transformers
Drew A Hudson and Larry Zitnick · 2021
Closest in time.
Deflocnet: Deep image editing via flexible low-level controls
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao, Bin Jiang, and Wei Liu · 2021
Closest in time.
Giraffe: Representing scenes as compositional generative neural feature fields
Michael Niemeyer and Andreas Geiger · 2021
Closest in time.
Zero-shot text-to-image generation
Original
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Closest in time.
Encoding in style: a stylegan encoder for image-to-image translation
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or · 2021
Closest in time.
Designing an encoder for stylegan image manipulation
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or · 2021
Closest in time.
High-fidelity pluralistic image completion with transformers
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao · 2021
Closest in time.
Diverse image inpainting with bidirectional and autoregressive transformers
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao · 2021
Closest in time.
BEit: BERT pre-training of image transformers
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei · 2022
Closest in time.
Masked autoencoders are scalable vision learners
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick · 2022
Closest in time.
SDEdit: Guided image synthesis and editing with stochastic differential equations
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…