Conformers are All You Need for Visual Speech Recognition

Understand

Visual speech recognition models extract visual features in a hierarchical manner.

• At the lower level, there is a visual front-end with a limited temporal receptive field that processes the raw pixels depicting the lips or faces.
• At the higher level, there is an encoder that attends to the embeddings produced by the front-end over a large temporal receptive field.
• Previous work has focused on improving the visual front-end of the model to extract more useful features for speech recognition.