2020

StyleRig: Rigging StyleGAN for 3D Control over Portrait Images

Tewari, Ayush, Elgharib, Mohamed, Bharaj, Gaurav et al.

Understand

StyleGAN generates photorealistic portrait images of faces with eyes, teeth, hair and context (neck, shoulders, background), but lacks a rig-like control over semantic face parameters that are interpretable in 3D, such as face pose, expressions, and scene illumination.

Three-dimensional morphable face models (3DMMs) on the other hand offer control over the semantic parameters, but lack photorealism when rendered and only model the face interior, not other parts of a portrait image (hair, mouth interior, background).
We present the first method to provide a face rig-like control over a pretrained and fixed StyleGAN via a 3DMM.
A new rigging network, RigNet is trained between the 3DMM's semantic parameters and StyleGAN's input.

Built on

A morphable model for the synthesis of 3D faces
Volker Blanz and Thomas Vetter · 1999
Earlier work this paper cites.
Reanimating faces in images and video
Volker Blanz, Curzio Basso, Tomaso Poggio, and Thomas Vetter · 2003
Earlier work this paper cites.
The digital emily project: Photoreal facial modeling and animation
Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec · 2009
Earlier work this paper cites.
Deformable model fitting by regularized landmark mean-shift
Jason M. Saragih, Simon Lucey, and Jeffrey F. Cohn · 2011
Earlier work this paper cites.
Adadelta: An adaptive learning rate method
Original
Matthew D. Zeiler · 2012
Earlier work this paper cites.
Facewarehouse: A 3D facial expression database for visual computing
Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou · 2014
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Real-time facial animation with image-based dynamic avatars
Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou · 2016
Earlier work this paper cites.
Reconstruction of personalized 3D face rigs from monocular video
Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt · 2016
Earlier work this paper cites.
Image style transfer using convolutional neural networks
L. A. Gatys, A. S. Ecker, and M. Bethge · 2016
Earlier work this paper cites.
Painting style transfer for head portraits using convolutional neural networks
Ahmed Selim, Mohamed Elgharib, and Linda Doyle · 2016
Earlier work this paper cites.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Earlier work this paper cites.
FLAME: Learning a model of facial shape and expression from 4d scans
Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero · 2017
Earlier work this paper cites.
Unsupervised image-to-image translation networks
Ming-Yu Liu, Thomas Breuel, and Jan Kautz · 2017
Earlier work this paper cites.

Similar

Learning detailed face reconstruction from a single image
Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel · 2017
Cited alongside, same era.
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation
Matan Sela, Elad Richardson, and Ron Kimmel · 2017
Cited alongside, same era.
MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian · 2017
Cited alongside, same era.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros · 2017
Cited alongside, same era.
Large scale 3D morphable models
James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou · 2018
Cited alongside, same era.
Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3d morphable model
Baris Gecer, Binod Bhattarai, Josef Kittler, and Tae-Kyun Kim · 2018
Cited alongside, same era.
Unsupervised training for 3d morphable model regression
Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T Freeman · 2018
Cited alongside, same era.
Progressive growing of GANs for improved quality, stability, and variation
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2018
Cited alongside, same era.
Deep Video Portraits
H. Kim, P. Garrido, A. Tewari, W. Xu, J. Thies, N. Nießner, P. Pérez, C. Richardt, M. Zollhöfer, and C. Theobalt · 2018
Cited alongside, same era.
InverseFaceNet: Deep Single-Shot Inverse Face Rendering From A Single Image
Hyeongwoo Kim, Michael Zollhöfer, Ayush Tewari, Justus Thies, Christian Richardt, and Christian Theobalt · 2018
Cited alongside, same era.
Lookingood: Enhancing performance capture with real-time neural re-rendering
Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, and Sean Fanello · 2018
Cited alongside, same era.
Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz
Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt · 2018
Cited alongside, same era.

Then

Everybody dance now
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros · 2019
Later among the works it cites.
Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong · 2019
Later among the works it cites.
3d morphable face models – past, present and future
Original
Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter · 2019
Later among the works it cites.
On the ”steerability” of generative adversarial networks
Original
Ali Jahanian, Lucy Chai, and Phillip Isola · 2019
Later among the works it cites.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Later among the works it cites.
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu · 2019
Later among the works it cites.
Learning to regress 3D face shape and expression from an image without 3D supervision
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael Black · 2019
Later among the works it cites.
Interpreting the latent space of gans for semantic face editing
Original
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou · 2019
Later among the works it cites.
Fml: Face model learning from videos
Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt · 2019
Later among the works it cites.
Deferred neural rendering: Image synthesis using neural textures
Justus Thies, Michael Zollhöfer, and Matthias Nießner · 2019
Later among the works it cites.
Towards high-fidelity nonlinear 3d face morphable model
Luan Tran, Feng Liu, and Xiaoming Liu · 2019
Later among the works it cites.
Puppetgan: Cross-domain image manipulation by demonstration
Ben Usman, Nick Dufour, Kate Saenko, and Chris Bregler · 2019
Later among the works it cites.
InverseRenderNet: Learning single image inverse rendering
Ye Yu and William A. P. Smith · 2019
Later among the works it cites.
State of the Art on Neural Rendering
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhöfer · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…