Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities

Understand

Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities.

• The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities.
• However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time.
• With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time.