Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss

Understand

We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions.

• Instead of learning a direct mapping from audio to video frames, we propose first to transfer audio to high-level structure, i.e., the facial landmarks, and then to generate video frames conditioned on the landmarks.
• Compared to a direct audio-to-image approach, our cascade approach avoids fitting spurious correlations between audiovisual signals that are irrelevant to the speech content.
• We, humans, are sensitive to temporal discontinuities and subtle artifacts in video.