2023

Control4D: Efficient 4D Portrait Editing with Text

Shao, Ruizhi, Sun, Jingxiang, Peng, Cheng et al.

Understand

We introduce Control4D, an innovative framework for editing dynamic 4D portraits using text instructions.

Our method addresses the prevalent challenges in 4D editing, notably the inefficiencies of existing 4D representations and the inconsistent editing effect caused by diffusion-based editors.
We first propose GaussianPlanes, a novel 4D representation that makes Gaussian Splatting more structured by applying plane-based decomposition in 3D space and time.
This enhances both efficiency and robustness in 4D editing.

Built on

Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Structure-from-motion revisited
Johannes Lutz Schönberger and Jan-Michael Frahm · 2016
Earlier work this paper cites.
Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum · 2016
Earlier work this paper cites.
Wasserstein generative adversarial networks
Martin Arjovsky, Soumith Chintala, and Léon Bottou · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam · 2017
Earlier work this paper cites.
High-resolution image synthesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro · 2018
Earlier work this paper cites.
Visual object networks: Image generation with disentangled 3d representations
Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman · 2018
Earlier work this paper cites.
Hologan: Unsupervised learning of 3d representations from natural images
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang · 2019
Earlier work this paper cites.
Generative adversarial networks
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
Blockgan: Learning 3d object-aware scene representations from unlabelled images
Thu H Nguyen-Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra · 2020
Earlier work this paper cites.
Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger · 2020
Earlier work this paper cites.
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol · 2021
Earlier work this paper cites.
Neural radiance flow for 4d view synthesis and video processing
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu · 2021
Earlier work this paper cites.
Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis
Original
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt · 2021
Earlier work this paper cites.
Variational diffusion models
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho · 2021
Earlier work this paper cites.
Neural actor: Neural free-view synthesis of human actors with pose control
Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt · 2021
Earlier work this paper cites.
Nerf: Representing scenes as neural radiance fields for view synthesis
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng · 2021
Earlier work this paper cites.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
Original
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen · 2021
Earlier work this paper cites.
Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou · 2021
Earlier work this paper cites.
D-nerf: Neural radiance fields for dynamic scenes
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Earlier work this paper cites.
Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt · 2021
Earlier work this paper cites.
Space-time neural irradiance fields for free-viewpoint video
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim · 2021
Earlier work this paper cites.
Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors
Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu · 2021
Earlier work this paper cites.

Similar

Text2live: Text-driven layered image and video editing
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel · 2022
Cited alongside, same era.
Instructpix2pix: Learning to follow image editing instructions
Original
Tim Brooks, Aleksander Holynski, and Alexei A Efros · 2022
Cited alongside, same era.
Efficient geometry-aware 3d generative adversarial networks
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al · 2022
Cited alongside, same era.
Vector quantized diffusion model for text-to-image synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo · 2022
Cited alongside, same era.
Classifier-free diffusion guidance
Original
Jonathan Ho and Tim Salimans · 2022
Cited alongside, same era.
Neural 3d video synthesis from multi-view video
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al · 2022
Cited alongside, same era.
Devrf: Fast deformable voxel radiance fields for dynamic scenes
Original
Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou · 2022
Cited alongside, same era.
Latent-nerf for shape-guided generation of 3d shapes and textures
Original
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or · 2022
Cited alongside, same era.
Instant neural graphics primitives with a multiresolution hash encoding
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller · 2022
Cited alongside, same era.
Dreamfusion: Text-to-3d using 2d diffusion
Original
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall · 2022
Cited alongside, same era.
Neuphysics: Editable neural geometry and physics from monocular videos
Yi-Ling Qiao, Alexander Gao, and Ming Lin · 2022
Cited alongside, same era.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Cited alongside, same era.

Then

Animatediff: Animate your personalized text-to-image diffusion models without specific tuning
Original
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai · 2023
Closest in time.
Instruct-nerf2nerf: Editing 3d scenes with instructions
Original
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa · 2023
Closest in time.
Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation
Original
Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang · 2023
Closest in time.
Humanrf: High-fidelity neural radiance fields for humans in motion
Original
Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner · 2023
Closest in time.
4d-editor: Interactive object-level editing in dynamic neural radiance fields via 4d semantic segmentation
Original
Dadong Jiang, Zhihui Ke, Xiaobo Zhou, and Xidong Shi · 2023
Closest in time.
3d gaussian splatting for real-time radiance field rendering
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis · 2023
Closest in time.
Text2video-zero: Text-to-image diffusion models are zero-shot video generators
Original
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi · 2023
Closest in time.
Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d
Original
Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan · 2023
Closest in time.
3d-clfusion: Fast text-to-3d rendering with contrastive latent diffusion
Original
Yu-Jhe Li and Kris Kitani · 2023
Closest in time.
Zero-1-to-3: Zero-shot one image to 3d object, 2023
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick · 2023
Closest in time.
Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis
Original
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan · 2023
Closest in time.
Compositional 3d scene generation using locally conditioned diffusion
Original
Ryan Po and Gordon Wetzstein · 2023
Closest in time.
Ditto-nerf: Diffusion-based iterative text to omni-directional 3d model
Original
Hoigi Seo, Hayeon Kim, Gwanghyun Kim, and Se Young Chun · 2023
Closest in time.
Realistic vision v5 stable diffusion checkpoint, 2023
SG_161222 · 2023
Closest in time.
Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu · 2023
Closest in time.
Anything-3d: Towards single-view anything reconstruction in the wild
Original
Qiuhong Shen, Xingyi Yang, and Xinchao Wang · 2023
Closest in time.
Mvdream: Multi-view diffusion for 3d generation
Original
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang · 2023
Closest in time.
Text-to-4d dynamic scene generation
Original
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al · 2023
Closest in time.
Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger · 2023
Closest in time.
Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior
Original
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu · 2023
Closest in time.
Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu · 2023
Closest in time.
4d gaussian splatting for real-time dynamic scene rendering
Original
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang · 2023
Closest in time.
4k4d: Real-time 4d view synthesis at 4k resolution
Original
Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou · 2023
Closest in time.
Adding conditional control to text-to-image diffusion models
Original
Lvmin Zhang and Maneesh Agrawala · 2023
Closest in time.
Avatarrex: Real-time expressive full-body avatars
Original
Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…