M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues

Understand

We present M3ER, a learning-based method for emotion recognition from multiple input modalities.

• Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities.
• M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a per-sample basis.
• By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise.