Affective Faces for Goal-Driven Dyadic Communication

Understand

We introduce a video framework for modeling the association between verbal and non-verbal communication during dyadic conversation.

• Given the input speech of a speaker, our approach retrieves a video of a listener, who has facial expressions that would be socially appropriate given the context.
• Our approach further allows the listener to be conditioned on their own goals, personalities, or backgrounds.
• Our approach models conversations through a composition of large language models and vision-language models, creating internal representations that are interpretable and controllable.