Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion

Understand

In this paper, we present a multi-modal online person verification system using both speech and visual signals.

• Inspired by neuroscientific findings on the association of voice and face, we propose an attention-based end-to-end neural network that learns multi-sensory associations for the task of person verification.
• The attention mechanism in our proposed network learns to conditionally select a salient modality between speech and facial representations that provides a balance between complementary inputs.
• By virtue of this capability, the network is robust to missing or corrupted data from either modality.