FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Understand

Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data.

• Prior works typically focus on learning phoneme-level features of short audio windows with limited context, occasionally resulting in inaccurate lip movements.
• To tackle this limitation, we propose a Transformer-based autoregressive model, FaceFormer, which encodes the long-term audio context and autoregressively predicts a sequence of animated 3D face meshes.
• To cope with the data scarcity issue, we integrate the self-supervised pre-trained speech representations.