Audio-Visual Speech Inpainting with Deep Learning

Understand

In this paper, we present a deep-learning-based framework for audio-visual speech inpainting, i.e., the task of restoring the missing parts of an acoustic speech signal from reliable audio context and uncorrupted visual information.

• Recent work focuses solely on audio-only methods and generally aims at inpainting music signals, which show highly different structure than speech.
• Instead, we inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration.
• We also experiment with a multi-task learning approach where a phone recognition task is learned together with speech inpainting.