Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

Understand

Articulation, emotion, and personality play strong roles in the orofacial movements.

• To improve the naturalness and expressiveness of virtual agents (VAs), it is important that we carefully model the complex interplay between these factors.
• This paper proposes a conditional generative adversarial network, called conditional sequential GAN (CSG), which learns the relationship between emotion and lexical content in a principled manner.
• This model uses a set of articulatory and emotional features directly extracted from the speech signal as conditioning inputs, generating realistic movements.