Cross-Modal Knowledge Transfer via Inter-Modal Translation and Alignment for Affect Recognition

Understand

Multi-modal affect recognition models leverage complementary information in different modalities to outperform their uni-modal counterparts.

• However, due to the unavailability of modality-specific sensors or data, multi-modal models may not be always employable.
• For this reason, we aim to improve the performance of uni-modal affect recognition models by transferring knowledge from a better-performing (or stronger) modality to a weaker modality during training.
• Our proposed multi-modal training framework for cross-modal knowledge transfer relies on two main steps.