2018

Text-to-image Synthesis via Symmetrical Distillation Networks

Yuan, Mingkuan, Peng, Yuxin

Understand

Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task.

The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps.
The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions.
For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g.

Built on

Multimedia content processing through cross-modal association. In Proceedings of the ACM on Multimedia Conference . 604–611
Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K Sethi. 2003 · 2003
Earlier work this paper cites.
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004 · 2004
Earlier work this paper cites.
Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM, 535–541
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006 · 2006
Earlier work this paper cites.
Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 722–729
Maria-Elena Nilsback and Andrew Zisserman. 2008 · 2008
Earlier work this paper cites.
Generating more realistic images using gated MRF’s. In Advances in Neural Information Processing Systems . 2002–2010
Volodymyr Mnih, Geoffrey E Hinton, et al · 2010
Earlier work this paper cites.
A new approach to cross-modal multimedia retrieval. In Proceedings of the ACM on Multimedia Conference . 251–260
Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. 2010 · 2010
Earlier work this paper cites.
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011 · 2011
Earlier work this paper cites.
FSIM: a feature similarity index for image quality assessment
Lin Zhang, Lei Zhang, Xuanqin Mou, David Zhang, et al · 2011
Earlier work this paper cites.
Multiple choice learning: Learning to produce multiple structured outputs. In Advances in Neural Information Processing Systems . 1799–1807
Abner Guzman-Rivera, Dhruv Batra, and Pushmeet Kohli. 2012 · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 1097–1105
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012 · 2012
Earlier work this paper cites.
Cross-modal retrieval with correspondence autoencoder. In Proceedings of the ACM on Multimedia Conference . 7–16
Fangxiang Feng, Xiaojie Wang, and Ruifan Li. 2014 · 2014
Earlier work this paper cites.
Generative adversarial nets. In Advances in Neural Information Processing Systems . 2672–2680
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Diederik P Kingma and Max Welling. 2014 · 2014
Earlier work this paper cites.
How transferable are features in deep neural networks?. In Advances in Neural Information Processing Systems . 3320–3328
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014 · 2014
Earlier work this paper cites.
Start from scratch: Towards automatically identifying, modeling, and naming visual attributes. In Proceedings of the ACM on Multimedia Conference . 187–196
Hanwang Zhang, Yang Yang, Huanbo Luan, Shuicheng Yang, and Tat-Seng Chua. 2014 · 2014
Earlier work this paper cites.

Similar

DRAW: A recurrent neural network for image generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015 · 2015
Cited alongside, same era.
Distilling the knowledge in a neural network
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik Kingma and Jimmy Ba. 2015 · 2015
Cited alongside, same era.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015 · 2015
Cited alongside, same era.
Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3431–3440
Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015 · 2015
Cited alongside, same era.
Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 5188–5196
Aravindh Mahendran and Andrea Vedaldi. 2015 · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. 2015 · 2015
Cited alongside, same era.
Inverting visual representations with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4829–4837
Alexey Dosovitskiy and Thomas Brox. 2016 · 2016
Cited alongside, same era.
Region-based convolutional networks for accurate object detection and segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2016 · 2016
Cited alongside, same era.
Cross modal distillation for supervision transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2827–2836
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. 2016 · 2016
Cited alongside, same era.
Generating images from captions with attention
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2016 · 2016
Cited alongside, same era.
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems . 3387–3395
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. 2016 · 2016
Cited alongside, same era.

Then

Improved techniques for training gans. In Advances in Neural Information Processing Systems . 2234–2242
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016 · 2016
Later among the works it cites.
Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2818–2826
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016 · 2016
Later among the works it cites.
Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems . 4790–4798
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al · 2016
Later among the works it cites.
Attribute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision . Springer, 776–791
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016 · 2016
Later among the works it cites.
Augmenting supervised neural networks with unsupervised objectives for large-scale image classification
Yuting Zhang, Kibok Lee, and Honglak Lee. 2016 · 2016
Later among the works it cites.
Towards principled methods for training generative adversarial networks
Martin Arjovsky and Léon Bottou. 2017 · 2017
Later among the works it cites.
BEGAN: Boundary equilibrium generative adversarial networks
Original
David Berthelot, Thomas Schumm, and Luke Metz. 2017 · 2017
Later among the works it cites.
Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE International Conference on Computer Vision . 1511–1520
Qifeng Chen and Vladlen Koltun. 2017 · 2017
Later among the works it cites.
Improved training of wasserstein GANs
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017 · 2017
Later among the works it cites.
Fine-grained image classification via combining vision and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 5994–6002
Xiangteng He and Yuxin Peng. 2017 · 2017
Later among the works it cites.
Stacked generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 1866–1875
Xun Huang, Yixuan Li, Omid Poursaeed, John E. Hopcroft, and Serge J. Belongie. 2017 · 2017
Later among the works it cites.
Pose guided person image generation
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. 2017 · 2017
Later among the works it cites.
Adversarial Cross-Modal Retrieval. In Proceedings of the ACM on Multimedia Conference . 154–162
Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017 · 2017
Later among the works it cites.
StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. 2017 · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…