Modality Distillation with Multiple Stream Networks for Action Recognition

Understand

Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance.

• However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities could be available in real life (testing) scenarios, where a model has to be deployed.
• This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities.
• This paper presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation.