Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Understand

Speech enhancement (SE) aims to reduce noise in speech signals.

• Most SE techniques focus only on addressing audio information.
• In this work, inspired by multimodal learning, which utilizes data from different modalities, and the recent success of convolutional neural networks (CNNs) in SE, we propose an audio-visual deep CNNs (AVDCNN) SE model, which incorporates audio and visual streams into a unified network model.
• We also propose a multi-task learning framework for reconstructing audio and visual signals at the output layer.