Deep Variational Generative Models for Audio-visual Speech Separation

Understand

In this paper, we are interested in audio-visual speech separation given a single-channel audio recording as well as visual information (lips movements) associated with each speaker.

• We propose an unsupervised technique based on audio-visual generative modeling of clean speech.
• More specifically, during training, a latent variable generative model is learned from clean speech spectrograms using a variational auto-encoder (VAE).
• To better utilize the visual information, the posteriors of the latent variables are inferred from mixed speech (instead of clean speech) as well as the visual data.