Multimodal End-to-End Sparse Model for Emotion Recognition

Understand

Existing works on multimodal affective computing tasks, such as emotion recognition, generally adopt a two-phase pipeline, first extracting feature representations for each single modality with hand-crafted algorithms and then performing end-to-end learning with the extracted features.

• However, the extracted features are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extraction algorithms does not generalize or scale well to different tasks, which can lead to sub-optimal performance.
• In this paper, we develop a fully end-to-end model that connects the two phases and optimizes them jointly.
• In addition, we restructure the current datasets to enable the fully end-to-end training.