Memory-augmented Contrastive Learning for Talking Head Generation

Understand

Given one reference facial image and a piece of speech as input, talking head generation aims to synthesize a realistic-looking talking head video.

• However, generating a lip-synchronized video with natural head movements is challenging.
• The same speech clip can generate multiple possible lip and head movements, that is, there is no one-to-one mapping relationship between them.
• To overcome this problem, we propose a Speech Feature Extractor (SFE) based on memory-augmented self-supervised contrastive learning, which introduces the memory module to store multiple different speech mapping results.