Fetching the paper…

Reading the bibliography…

2023

MagicAvatar: Multimodal Avatar Generation and Animation

Zhang, Jianfeng, Yan, Hanshu, Xu, Zhongcong et al.

Original
PDF
Post on X

Understand

This report presents MagicAvatar, a framework for multimodal video generation and animation of human avatars.

Unlike most existing methods that generate avatar-centric videos directly from multimodal inputs (e.g., text prompts), MagicAvatar explicitly disentangles avatar video generation into two stages: (1) multimodal-to-motion and (2) motion-to-video generation.
The first stage translates the multimodal inputs into motion/ control signals (e.g., human pose, depth, DensePose); while the second stage generates avatar-centric video guided by these motion signals.
Additionally, MagicAvatar supports avatar animation by simply providing a few images of the target person.

MagicAvatar: Multimodal Avatar Generation and Animation · Around

Built on

Realtime multi-person 2d pose estimation using part affinity fields
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh · 2017
Earlier work this paper cites.
Densepose: Dense human pose estimation in the wild
Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos · 2018
Earlier work this paper cites.

Similar

Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun · 2020
Cited alongside, same era.
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman · 2022
Cited alongside, same era.

Then

Magicedit: High-fidelity and temporally coherent video editing
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng · 2023
Closest in time.
Human motion diffusion model
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…