ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization

Understand

Modality selection is an important step when designing multimodal systems, especially in the case of cross-domain activity recognition as certain modalities are more robust to domain shift than others.

• However, selecting only the modalities which have a positive contribution requires a systematic approach.
• We tackle this problem by proposing an unsupervised modality selection method (ModSelect), which does not require any ground-truth labels.
• We determine the correlation between the predictions of multiple unimodal classifiers and the domain discrepancy between their embeddings.