Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templates

Understand

Co-speech gesture generation is to synthesize a gesture sequence that not only looks real but also matches with the input speech audio.

• Our method generates the movements of a complete upper body, including arms, hands, and the head.
• Although recent data-driven methods achieve great success, challenges still exist, such as limited variety, poor fidelity, and lack of objective metrics.
• Motivated by the fact that the speech cannot fully determine the gesture, we design a method that learns a set of gesture template vectors to model the latent conditions, which relieve the ambiguity.