2018

Everybody Dance Now

Chan, Caroline, Ginosar, Shiry, Zhou, Tinghui et al.

Understand

This paper presents a simple method for "do as I do" motion transfer: given a source video of a person dancing, we can transfer that performance to a novel (amateur) target after only a few minutes of the target subject performing standard moves.

We approach this problem as video-to-video translation using pose as an intermediate representation.
To transfer the motion, we extract poses from the source subject and apply the learned pose-to-appearance mapping to generate the target subject.
We predict two consecutive frames for temporally coherent video results and introduce a separate pipeline for realistic face synthesis.

Built on

Video rewrite: Driving visual speech with audio
Christoph Bregler, Michele Covell, and Malcolm Slaney · 1997
Earlier work this paper cites.
Retargetting motion to new characters
Michael Gleicher · 1998
Earlier work this paper cites.
A hierarchical approach to interactive motion editing for human-like figures
Jehee Lee and Sung Yong Shin · 1999
Earlier work this paper cites.
Recognizing action at a distance
Alexei A. Efros, Alexander C. Berg, Greg Mori, and Jitendra Malik · 2003
Earlier work this paper cites.
Markerless human motion transfer
German KM Cheung, Simon Baker, Jessica Hodgins, and Takeo Kanade · 2004
Earlier work this paper cites.
Video based motion synthesis by splicing and morphing
Greg Mori, Alex Berg, Alexei Efros, Ashley Eden, and Jitendra Malik · 2004
Earlier work this paper cites.
Image quality assessment: from error visibility to structural similarity
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli · 2004
Earlier work this paper cites.
Real-time motion retargeting to highly varied user-created morphologies
Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen · 2008
Earlier work this paper cites.
Video-based characters: creating new human performances from a multi-view video database
Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gaurav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz, and Christian Theobalt · 2011
Earlier work this paper cites.
4D Video Textures for Interactive Character Appearance
Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Earlier work this paper cites.
Perceptual losses for real-time style transfer and super-resolution
Justin Johnson, Alexandre Alahi, and Li Fei-Fei · 2016
Earlier work this paper cites.
Coupled generative adversarial networks
Ming-Yu Liu and Oncel Tuzel · 2016
Earlier work this paper cites.
Convolutional pose machines
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh · 2016
Earlier work this paper cites.
Dynamics transfer gan: Generating video by transferring arbitrary temporal dynamics from a source video to a single target image
Original
Wissam J Baddar, Geonmo Gu, Sangmin Lee, and Yong Man Ro · 2017
Earlier work this paper cites.
Realtime multi-person 2D pose estimation using part affinity fields
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh · 2017
Earlier work this paper cites.

Similar

Photographic image synthesis with cascaded refinement networks
Qifeng Chen and Vladlen Koltun · 2017
Cited alongside, same era.
Image-to-image translation with conditional adversarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros · 2017
Cited alongside, same era.
Learning to discover cross-domain relations with generative adversarial networks
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim · 2017
Cited alongside, same era.
Understanding black-box predictions via influence functions
Pang Wei Koh and Percy Liang · 2017
Cited alongside, same era.
A generative model of people in clothing
Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler · 2017
Cited alongside, same era.
Unsupervised image-to-image translation networks
Ming-Yu Liu, Thomas Breuel, and Jan Kautz · 2017
Cited alongside, same era.
Pose guided person image generation
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool · 2017
Cited alongside, same era.
Hand keypoint detection in single images using multiview bootstrapping
Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh · 2017
Cited alongside, same era.
Learning to generate long-term future via hierarchical prediction
Original
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee · 2017
Cited alongside, same era.
The pose knows: Video forecasting by generating pose futures
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert · 2017
Cited alongside, same era.
Unpaired image-to-image translation using cycle-consistent adversarial networkss
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros · 2017
Cited alongside, same era.
Synthesizing images of humans in unseen poses
Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag · 2018
Cited alongside, same era.

Then

Deep video portraits
Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt · 2018
Closest in time.
Disentangled person image generation
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz · 2018
Closest in time.
Lookingood: Enhancing performance capture with real-time neural re-rendering
Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, and Sean Fanello · 2018
Closest in time.
Densepose: Dense human pose estimation in the wild
Iasonas Kokkinos Rıza Alp Güler, Natalia Neverova · 2018
Closest in time.
Deformable gans for pose-based human image generation
Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe · 2018
Closest in time.
Mocogan: Decomposing motion and content for video generation
Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz · 2018
Closest in time.
Neural kinematic networks for unsupervised motion retargetting
Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee · 2018
Closest in time.
Video-to-video synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro · 2018
Closest in time.
High-resolution image synthesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro · 2018
Closest in time.
Human appearance transfer
Mihai Zanfir, Alin-Ionut Popa, Andrei Zanfir, and Cristian Sminchisescu · 2018
Closest in time.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Closest in time.
Deep video-based performance cloning
Kfir Aberman, Mingyi Shi, Jing Liao, D Liscbinski, Baoquan Chen, and Daniel Cohen-Or · 2019
Closest in time.
A conditional deep generative model of people in natural images
Rodrigo De Bem, Arnab Ghosh, Adnane Boukhayma, Thalaiyasingam Ajanthan, N Siddharth, and Philip Torr · 2019
Closest in time.
Neural rendering and reenactment of human actor videos
Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Habermann, Wenping Wang, and Christian Theobalt · 2019
Closest in time.
Dance x Machine Learning: First Steps
Kyle McDonald · 2019
Closest in time.
Using AI to make NBA players dance
Xpire · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…