Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping

Understand

Visual Speech Recognition (VSR) differs from the common perception tasks as it requires deeper reasoning over the video sequence, even by human experts.

• Despite the recent advances in VSR, current approaches rely on labeled data to fully train or finetune their models predicting the target speech.
• This hinders their ability to generalize well beyond the training set and leads to performance degeneration under out-of-distribution challenging scenarios.
• Unlike previous works that involve auxiliary losses or complex training procedures and architectures, we propose a simple approach, named Lip2Vec that is based on learning a prior model.