One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

Understand

Audio-driven one-shot talking face generation methods are usually trained on video resources of various persons.

• However, their created videos often suffer unnatural mouth shapes and asynchronous lips because those methods struggle to learn a consistent speech style from different speakers.
• We observe that it would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements.
• Hence, we propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker and then transferring audio-driven motion fields to a reference image.