Fetching the paper…

Reading the bibliography…

2021

VirtualConductor: Music-driven Conducting Video Generation System

Chen, Delong, Liu, Fan, Li, Zewen et al.

Original
PDF
Post on X

Understand

In this demo, we present VirtualConductor, a system that can generate conducting video from any given music and a single user's image.

First, a large-scale conductor motion dataset is collected and constructed.
Then, we propose Audio Motion Correspondence Network (AMCNet) and adversarial-perceptual learning to learn the cross-modal relationship and generate diverse, plausible, music-synchronized motion.
Finally, we combine 3D animation rendering and a pose transfer model to synthesize conducting video from a single given user's image.

VirtualConductor: Music-driven Conducting Video Generation System · Around

Built on

“Rmpe: Regional multi-person pose estimation,”
H. Fang, S. Xie, Y. Tai, and C. Lu, · 2017
Earlier work this paper cites.
“Deep collaborative embedding for social image understanding,”
Zechao Li, Jinhui Tang, and Tao Mei, · 2019
Earlier work this paper cites.

Similar

“Show, reward, and tell: Adversarial visual story generation,”
Jinhui Tang, Jing Wang, Zechao Li, Jianlong Fu, and Tao Mei, · 2019
Cited alongside, same era.

Then

“Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis,”
Wen Liu, Wenhan Luo Lin Ma Zhixin Piao, Min Jie, and Shenghua Gao, · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…