Dynamic Temporal Alignment of Speech to Lips

Understand

Many speech segments in movies are re-recorded in a studio during postproduction, to compensate for poor sound quality as recorded on location.

• Manual alignment of the newly-recorded speech with the original lip movements is a tedious task.
• We present an audio-to-video alignment method for automating speech to lips alignment, stretching and compressing the audio signal to match the lip movements.
• This alignment is based on deep audio-visual features, mapping the lips video and the speech signal to a shared representation.