Multi Modal Facial Expression Recognition with Transformer-Based Fusion Networks and Dynamic Sampling

Understand

Facial expression recognition is an essential task for various applications, including emotion detection, mental health analysis, and human-machine interactions.

• In this paper, we propose a multi-modal facial expression recognition method that exploits audio information along with facial images to provide a crucial clue to differentiate some ambiguous facial expressions.
• Specifically, we introduce a Modal Fusion Module (MFM) to fuse audio-visual information, where image and audio features are extracted from Swin Transformer.
• Additionally, we tackle the imbalance problem in the dataset by employing dynamic data resampling.