Look, Listen and Learn - A Multimodal LSTM for Speaker Identification

Understand

Speaker identification refers to the task of localizing the face of a person who has the same identity as the ongoing voice in a video.

• This task not only requires collective perception over both visual and auditory signals, the robustness to handle severe quality degradations and unconstrained content variations are also indispensable.
• In this paper, we describe a novel multimodal Long Short-Term Memory (LSTM) architecture which seamlessly unifies both visual and auditory modalities from the beginning of each sequence input.
• The key idea is to extend the conventional LSTM by not only sharing weights across time steps, but also sharing weights across modalities.