PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis

Understand

Talking head synthesis with arbitrary speech audio is a crucial challenge in the field of digital humans.

• Recently, methods based on radiance fields have received increasing attention due to their ability to synthesize high-fidelity and identity-consistent talking heads from just a few minutes of training video.
• However, due to the limited scale of the training data, these methods often exhibit poor performance in audio-lip synchronization and visual quality.
• In this paper, we propose a novel 3D Gaussian-based method called PointTalk, which constructs a static 3D Gaussian field of the head and deforms it in sync with the audio.