2017

Demystifying Neural Style Transfer

Li, Yanghao, Wang, Naiyan, Liu, Jiaying et al.

Understand

Neural Style Transfer has recently demonstrated very exciting results which catches eyes in both academia and industry.

Despite the amazing results, the principle of neural style transfer, especially why the Gram matrices could represent style remains unclear.
In this paper, we propose a novel interpretation of neural style transfer by treating it as a domain adaptation problem.
Specifically, we theoretically show that matching the Gram matrices of feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel.

Built on

Texture synthesis by non-parametric sampling
Alexei A Efros and Thomas K Leung · 1999
Earlier work this paper cites.
Image quilting for texture synthesis and transfer
Alexei A Efros and William T Freeman · 2001
Earlier work this paper cites.
Image analogies
Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin · 2001
Earlier work this paper cites.
Real-time texture synthesis by patch-based sampling
Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung-Yeung Shum · 2001
Earlier work this paper cites.
Texture optimization for example-based synthesis
Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra · 2005
Earlier work this paper cites.
A survey on transfer learning
Sinno Jialin Pan and Qiang Yang · 2010
Earlier work this paper cites.
Domain adaptations for computer vision applications
Original
Oscar Beijbom · 2012
Earlier work this paper cites.
A kernel two-sample test
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola · 2012
Earlier work this paper cites.
Optimal kernel choice for large-scale two-sample tests
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur · 2012
Earlier work this paper cites.

Similar

Style transfer for headshot portraits
YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand · 2014
Cited alongside, same era.
Deep domain confusion: Maximizing for domain invariance
Original
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell · 2014
Cited alongside, same era.
Learning transferable features with deep adaptation networks
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan · 2015
Cited alongside, same era.
Visual domain adaptation: A survey of recent advances
Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman · 2015
Cited alongside, same era.
Semantic style transfer and turning two-bit doodles into fine artworks
Original
Alex J Champandard · 2016
Cited alongside, same era.
MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang · 2016
Cited alongside, same era.
Split and match: Example-based adaptive patch sampling for unsupervised style transfer
Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier · 2016
Cited alongside, same era.
Image style transfer using convolutional neural networks
Leon A Gatys, Alexander S Ecker, and Matthias Bethge · 2016
Cited alongside, same era.

Then

Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei · 2016
Later among the works it cites.
Photo-realistic single image super-resolution using a generative adversarial network
Original
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi · 2016
Later among the works it cites.
Combining Markov random fields and convolutional neural networks for image synthesis
Chuan Li and Michael Wand · 2016
Later among the works it cites.
Unsupervised domain adaptation with residual transfer networks
Mingsheng Long, Jianmin Wang, and Michael I Jordan · 2016
Later among the works it cites.
Artistic style transfer for videos
Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox · 2016
Later among the works it cites.
Painting style transfer for head portraits using convolutional neural networks
Ahmed Selim, Mohamed Elgharib, and Linda Doyle · 2016
Later among the works it cites.
Return of frustratingly easy domain adaptation
Baochen Sun, Jiashi Feng, and Kate Saenko · 2016
Later among the works it cites.
Texture networks: Feed-forward synthesis of textures and stylized images
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky · 2016
Later among the works it cites.
Revisiting batch normalization for practical domain adaptation
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…