Fetching the paper…

Reading the bibliography…

2018

An Introduction to Image Synthesis with Generative Adversarial Nets

Huang, He, Yu, Philip S., Wang, Changhu

Original
PDF
Post on X

Understand

There has been a drastic growth of research in Generative Adversarial Nets (GANs) in the past few years.

Proposed in 2014, GAN has been applied to various applications such as computer vision and natural language processing, and achieves impressive performance.
Among the many applications of GAN, image synthesis is the most well-studied one, and research in this area has already demonstrated the great potential of using GAN in image synthesis.
In this paper, we provide a taxonomy of methods used in image synthesis, review different models for text-to-image synthesis and image-to-image translation, and discuss some evaluation metrics as well as possible future research directions in image synthesis with GAN.

An Introduction to Image Synthesis with Generative Adversarial Nets · Around

Built on

P. J. Burt and E. H. Adelson, “The laplacian pyramid as a compact image code,” in Readings in Computer Vision . Elsevier, 1987, pp. 671–679
1987
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
D. G. Lowe, “Object recognition from local scale-invariant features,” in Computer vision, 1999. The proceedings of the seventh IEEE international conference on , vol. 2. Ieee, 1999, pp. 1150–1157
1999
Earlier work this paper cites.
N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on , vol. 1. IEEE, 2005, pp. 886–893
2005
Earlier work this paper cites.
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in Proceedings of the 25th international conference on Machine learning . ACM, 2008, pp. 1096–1103
2008
Earlier work this paper cites.
M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008
2008
Earlier work this paper cites.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011
2011
Earlier work this paper cites.
G. E. Hinton, A. Krizhevsky, and S. D. Wang, “Transforming auto-encoders,” in International Conference on Artificial Neural Networks . Springer, 2011, pp. 44–51
2011
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in NIPS workshop on deep learning and unsupervised feature learning , vol. 2011, no. 2, 2011, p. 5
2011
Earlier work this paper cites.
T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning , vol. 4, no. 2, pp. 26–31, 2012
2012
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in neural information processing systems , 2014, pp. 2672–2680
2014
Earlier work this paper cites.
M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 , 2014
Original
2014
Earlier work this paper cites.
M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in Proceedings of the IEEE Conference on computer Vision and Pattern Recognition , 2014, pp. 3686–3693
2014
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision . Springer, 2014, pp. 740–755
2014
Earlier work this paper cites.
D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in Advances in neural information processing systems , 2014, pp. 2366–2374
2014
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434 , 2015
Original
2015
Earlier work this paper cites.
A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” arXiv preprint arXiv:1512.09300 , 2015
Original
2015
Earlier work this paper cites.
E. L. Denton, S. Chintala, a. szlam, and R. Fergus, “Deep generative image models using a laplacian pyramid of adversarial networks,” in Advances in Neural Information Processing Systems 28 . Curran Associates, Inc., 2015, pp. 1486–1494
2015
Earlier work this paper cites.
K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, “Draw: A recurrent neural network for image generation,” arXiv preprint arXiv:1502.04623 , 2015
Original
2015
Earlier work this paper cites.
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 2625–2634
2015
Earlier work this paper cites.
Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in Proceedings of International Conference on Computer Vision (ICCV) , 2015
2015
Earlier work this paper cites.
F. Yu, Y. Zhang, S. Song, A. Seff, and J. Xiao, “Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365 , 2015
Original
2015
Earlier work this paper cites.
H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” CoRR , vol. abs/1511.08861, 2015. [Online]. Available: http://arxiv.org/abs/1511.08861
Original
2015
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2015, pp. 234–241
2015
Earlier work this paper cites.
A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 5188–5196
2015
Earlier work this paper cites.
M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 , 2015
Original
2015
Earlier work this paper cites.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al. , “Mastering the game of go with deep neural networks and tree search,” Nature , vol. 529, no. 7587, pp. 484–489, 2016
2016
Earlier work this paper cites.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in Advances in Neural Information Processing Systems , 2016, pp. 2226–2234
2016
Earlier work this paper cites.
T. Che, Y. Li, A. P. Jacob, Y. Bengio, and W. Li, “Mode regularized generative adversarial networks,” arXiv preprint arXiv:1612.02136 , 2016
Original
2016
Earlier work this paper cites.
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in Advances In Neural Information Processing Systems , 2016, pp. 2172–2180
2016
Earlier work this paper cites.
J. Zhao, M. Mathieu, and Y. LeCun, “Energy-based generative adversarial network,” arXiv preprint arXiv:1609.03126 , 2016
Original
2016
Earlier work this paper cites.

Similar

I. Goodfellow, “Nips 2016 tutorial: Generative adversarial networks,” arXiv preprint arXiv:1701.00160 , 2016
Original
2016
Cited alongside, same era.
R. A. Yeh, C. Chen, T. Lim, M. Hasegawa-Johnson, and M. N. Do, “Semantic image inpainting with perceptual and contextual losses,” CoRR , vol. abs/1607.07539, 2016. [Online]. Available: http://arxiv.org/abs/1607.07539
Original
2016
Cited alongside, same era.
P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” arXiv preprint arXiv:1611.08408 , 2016
Original
2016
Cited alongside, same era.
V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv preprint arXiv:1603.07285 , 2016
Original
2016
Cited alongside, same era.
H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” arXiv preprint arXiv:1612.03242 , 2016

Then

N. Brown and T. Sandholm, “Safe and nested subgame solving for imperfect-information games,” arXiv preprint arXiv:1705.02955 , 2017
Original
2017
Later among the works it cites.
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv preprint arXiv:1701.07875 , 2017
Original
2017
Later among the works it cites.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gan,” arXiv preprint arXiv:1704.00028 , 2017
Original
2017
Later among the works it cites.
D. Berthelot, T. Schumm, and L. Metz, “Began: Boundary equilibrium generative adversarial networks,” arXiv preprint arXiv:1703.10717 , 2017
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2016

Cited alongside, same era.

S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” arXiv preprint arXiv:1605.05396 , 2016

2016

Cited alongside, same era.

S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, “Learning what and where to draw,” in Advances in Neural Information Processing Systems , 2016, pp. 217–225

2016

Cited alongside, same era.

A. Odena, “Semi-supervised learning with generative adversarial networks,” arXiv preprint arXiv:1606.01583 , 2016

2016

Cited alongside, same era.

A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier gans,” arXiv preprint arXiv:1610.09585 , 2016

2016

Cited alongside, same era.

D. Yoo, N. Kim, S. Park, A. S. Paek, and I. S. Kweon, “Pixel-level domain transfer,” in European Conference on Computer Vision . Springer, 2016, pp. 517–532

2016

Cited alongside, same era.

J. Donahue, P. Krähenbühl, and T. Darrell, “Adversarial feature learning,” arXiv preprint arXiv:1605.09782 , 2016

2016

Cited alongside, same era.

V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A. Courville, “Adversarially learned inference,” arXiv preprint arXiv:1606.00704 , 2016

2016

Cited alongside, same era.

L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative adversarial nets with policy gradient.” in AAAI , 2017, pp. 2852–2858

2017

Later among the works it cites.

A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” arXiv preprint arXiv:1710.07035 , 2017

2017

Later among the works it cites.

T.-H. Chen, Y.-H. Liao, C.-Y. Chuang, W.-T. Hsu, J. Fu, and M. Sun, “Show, adapt and tell: Adversarial training of cross-domain image captioner,” arXiv preprint arXiv:1705.00930 , 2017

2017

Later among the works it cites.

X. Liang, Z. Hu, H. Zhang, C. Gan, and E. P. Xing, “Recurrent topic-transition gan for visual paragraph generation,” arXiv preprint arXiv:1703.07022 , 2017

2017

Later among the works it cites.

W. Zhao, W. Xu, M. Yang, J. Ye, Z. Zhao, Y. Feng, and Y. Qiao, “Dual learning for cross-domain image captioning,” in CIKM , 11 2017, pp. 29–38

2017

Later among the works it cites.

X. Wang, A. Shrivastava, and A. Gupta, “A-fast-rcnn: Hard positive generation via adversary for object detection,” arXiv preprint arXiv:1704.03414 , 2017

2017

Later among the works it cites.

S. Rajeswar, S. Subramanian, F. Dutil, C. J. Pal, and A. C. Courville, “Adversarial generation of natural language,” CoRR , vol. abs/1705.10929, 2017. [Online]. Available: http://arxiv.org/abs/1705.10929

2017

Later among the works it cites.

J. Guo, S. Lu, H. Cai, W. Zhang, Y. Yu, and J. Wang, “Long text generation via adversarial training with leaked information,” CoRR , vol. abs/1709.08624, 2017. [Online]. Available: http://arxiv.org/abs/1709.08624

2017

Later among the works it cites.

J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky, “Adversarial learning for neural dialogue generation,” arXiv preprint arXiv:1701.06547 , 2017

2017

Later among the works it cites.

Z. Yang, J. Hu, R. Salakhutdinov, and W. W. Cohen, “Semi-supervised qa with generative domain-adaptive nets,” arXiv preprint arXiv:1702.02206 , 2017

2017

Later among the works it cites.

Z. Yang, W. Chen, F. Wang, and B. Xu, “Improving neural machine translation with conditional sequence generative adversarial nets,” arXiv preprint arXiv:1703.04887 , 2017

2017

Later among the works it cites.

A. Dash, J. C. B. Gamboa, S. Ahmed, M. Z. Afzal, and M. Liwicki, “Tac-gan-text conditioned auxiliary classifier generative adversarial network,” arXiv preprint arXiv:1703.06412 , 2017

2017

Later among the works it cites.

J. Yang, A. Kannan, D. Batra, and D. Parikh, “Lr-gan: Layered recursive generative adversarial networks for image generation,” arXiv preprint arXiv:1703.01560 , 2017

2017

Later among the works it cites.

T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196 , 2017

2017

Later among the works it cites.

H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan++: Realistic image synthesis with stacked generative adversarial networks,” CoRR , vol. abs/1710.10916, 2017. [Online]. Available: http://arxiv.org/abs/1710.10916

2017

Later among the works it cites.

T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” arXiv preprint arXiv:1711.10485 , 2017

Original

2017

Later among the works it cites.

S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” arXiv preprint arXiv:1710.09829v2 , 2017

2017

Later among the works it cites.

Y. Jing, Y. Yang, Z. Feng, J. Ye, and M. Song, “Neural style transfer: A review,” CoRR , vol. abs/1705.04058, 2017. [Online]. Available: http://arxiv.org/abs/1705.04058

2017

Later among the works it cites.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” arXiv preprint arXiv:1703.10593 , 2017

2017

Later among the works it cites.

Z. Yi, H. Zhang, P. T. Gong et al. , “Dualgan: Unsupervised dual learning for image-to-image translation,” arXiv preprint arXiv:1704.02510 , 2017

2017

Later among the works it cites.

T. Kim, M. Cha, H. Kim, J. Lee, and J. Kim, “Learning to discover cross-domain relations with generative adversarial networks,” arXiv preprint arXiv:1703.05192 , 2017

2017

Later among the works it cites.

X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” in 2017 IEEE International Conference on Computer Vision (ICCV) . IEEE, 2017, pp. 2813–2821

2017

Later among the works it cites.

S. Benaim and L. Wolf, “One-sided unsupervised domain mapping,” arXiv preprint arXiv:1706.00826 , 2017

Original

2017

Later among the works it cites.

M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in Neural Information Processing Systems , 2017, pp. 700–708

2017

Later among the works it cites.

Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” arXiv preprint arXiv:1711.09020 , 2017

2017

Later among the works it cites.

R. Huang, S. Zhang, T. Li, and R. He, “Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis,” CoRR , vol. abs/1704.04086, 2017. [Online]. Available: http://arxiv.org/abs/1704.04086

2017

Later among the works it cites.

M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet, “Are GANs Created Equal? A Large-Scale Study,” ArXiv e-prints , Nov. 2017

2017

Later among the works it cites.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a nash equilibrium,” CoRR , vol. abs/1706.08500, 2017. [Online]. Available: http://arxiv.org/abs/1706.08500

2017

Later among the works it cites.

W. Fedus, M. Rosca, B. Lakshminarayanan, A. M. Dai, S. Mohamed, and I. Goodfellow, “Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step,” ArXiv e-prints , Oct. 2017

2017

Later among the works it cites.

M. Jaderberg, K. Simonyan, A. Zisserman et al. , “Spatial transformer networks,” in Advances in Neural Information Processing Systems , 2015, pp. 2017–2025

2025

Closest in time.