2017

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Wang, Ting-Chun, Liu, Ming-Yu, Zhu, Jun-Yan et al.

Understand

We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs).

Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic.
In this work, we generate 2048x1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures.
Furthermore, we extend our framework to interactive visual manipulation with two additional features.

Built on

The Laplacian pyramid as a compact image code
P. Burt and E. Adelson · 1983
Earlier work this paper cites.
Nonlinear total variation based noise removal algorithms
L. I. Rudin, S. Osher, and E. Fatemi · 1992
Earlier work this paper cites.
Recognising panoramas
M. Brown, D. G. Lowe, et al · 2003
Earlier work this paper cites.
Semantic photo synthesis
M. Johnson, G. J. Brostow, J. Shotton, O. Arandjelovic, V. Kwatra, and R. Cipolla · 2006
Earlier work this paper cites.
Photo clip art
J.-F. Lalonde, D. Hoiem, A. A. Efros, C. Rother, J. Winn, and A. Criminisi · 2007
Earlier work this paper cites.
Sketch2photo: Internet image montage
T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu · 2009
Earlier work this paper cites.
Interactive facial feature localization
V. Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from RGBD images
P. K. Nathan Silberman, Derek Hoiem and R. Fergus · 2012
Earlier work this paper cites.
Auto-encoding variational Bayes
D. P. Kingma and M. Welling · 2013
Earlier work this paper cites.
Exemplar-based face parsing
B. M. Smith, L. Zhang, J. Brandt, Z. Lin, and J. Yang · 2013
Earlier work this paper cites.
Generative adversarial networks
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Earlier work this paper cites.
Conditional generative adversarial nets
Original
M. Mirza and S. Osindero · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and A. Zisserman · 2014
Earlier work this paper cites.
Deep generative image models using a Laplacian pyramid of adversarial networks
E. Denton, S. Chintala, A. Szlam, and R. Fergus · 2015
Earlier work this paper cites.
Fully convolutional networks for semantic segmentation
J. Long, E. Shelhamer, and T. Darrell · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
A. Radford, L. Metz, and S. Chintala · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
O. Ronneberger, P. Fischer, and T. Brox · 2015
Earlier work this paper cites.
The Cityscapes dataset for semantic urban scene understanding
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele · 2016
Earlier work this paper cites.
Generating images with perceptual similarity metrics based on deep networks
A. Dosovitskiy and T. Brox · 2016
Earlier work this paper cites.
Generative multi-adversarial networks
I. Durugkar, I. Gemp, and S. Mahadevan · 2016
Earlier work this paper cites.
Image style transfer using convolutional neural networks
L. A. Gatys, A. S. Ecker, and M. Bethge · 2016
Earlier work this paper cites.

Similar

Deep joint demosaicking and denoising
M. Gharbi, G. Chaurasia, S. Paris, and F. Durand · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Perceptual losses for real-time style transfer and super-resolution
J. Johnson, A. Alahi, and L. Fei-Fei · 2016
Cited alongside, same era.
Learning to generate images of outdoor scenes from attributes and semantic layouts
Original
L. Karacan, Z. Akata, A. Erdem, and E. Erdem · 2016
Cited alongside, same era.
Autoencoding beyond pixels using a learned similarity metric
A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther · 2016
Cited alongside, same era.
Coupled generative adversarial networks
M.-Y. Liu and O. Tuzel · 2016
Cited alongside, same era.
Deep multi-scale video prediction beyond mean square error
M. Mathieu, C. Couprie, and Y. LeCun · 2016
Cited alongside, same era.
Context encoders: Feature learning by inpainting
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros · 2016
Cited alongside, same era.
Improved techniques for training GANs
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen · 2016
Cited alongside, same era.
Scribbler: Controlling deep image synthesis with sketch and color
Original
P. Sangkloy, J. Lu, C. Fang, F. Yu, and J. Hays · 2016
Cited alongside, same era.
Instance normalization: The missing ingredient for fast stylization
Original
D. Ulyanov, A. Vedaldi, and V. Lempitsky · 2016
Cited alongside, same era.
Generating videos with scene dynamics
C. Vondrick, H. Pirsiavash, and A. Torralba · 2016
Cited alongside, same era.

Then

Stacked generative adversarial networks
X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie · 2017
Closest in time.
Globally and locally consistent image completion
S. Iizuka, E. Simo-Serra, and H. Ishikawa · 2017
Closest in time.
Image-to-image translation with conditional adversarial networks
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros · 2017
Closest in time.
Generative attribute controller with conditional filtered generative adversarial networks
T. Kaneko, K. Hiramatsu, and K. Kashino · 2017
Closest in time.
Progressive growing of gans for improved quality, stability, and variation
Original
T. Karras, T. Aila, S. Laine, and J. Lehtinen · 2017
Closest in time.
Photo-realistic single image super-resolution using a generative adversarial network
C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al · 2017
Closest in time.
Perceptual generative adversarial networks for small object detection
J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan · 2017
Closest in time.
Unsupervised image-to-image translation networks
M.-Y. Liu, T. Breuel, and J. Kautz · 2017
Closest in time.
Least squares generative adversarial networks
X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley · 2017
Closest in time.
Learning from simulated and unsupervised images through adversarial training
A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb · 2017
Closest in time.
Unsupervised cross-domain image generation
Y. Taigman, A. Polyak, and L. Wolf · 2017
Closest in time.
Mocogan: Decomposing motion and content for video generation
Original
S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz · 2017
Closest in time.
Adversarial inverse graphics networks: Learning 2D-to-3D lifting and image-to-image translation from unpaired supervision
H.-Y. F. Tung, A. W. Harley, W. Seto, and K. Fragkiadaki · 2017
Closest in time.
DualGAN: Unsupervised dual learning for image-to-image translation
Z. Yi, H. Zhang, P. T. Gong, et al · 2017
Closest in time.
StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks
H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas · 2017
Closest in time.
Real-time user-guided image colorization with learned deep priors
R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros · 2017
Closest in time.
Age progression/regression by conditional adversarial autoencoder
Z. Zhang, Y. Song, and H. Qi · 2017
Closest in time.
Pyramid scene parsing network
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia · 2017
Closest in time.
Energy-based generative adversarial network
J. Zhao, M. Mathieu, and Y. LeCun · 2017
Closest in time.
Scene parsing through ADE20K dataset
B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba · 2017
Closest in time.
Unpaired image-to-image translation using cycle-consistent adversarial networks
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros · 2017
Closest in time.
Toward multimodal image-to-image translation
J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…