Investigating Multilingual NMT Representations at Scale

Understand

Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings.

• However, these black-box representations are poorly understood, and their mode of transfer remains elusive.
• In this work, we attempt to understand massively multilingual NMT representations (with 103 languages) using Singular Value Canonical Correlation Analysis (SVCCA), a representation similarity framework that allows us to compare representations across different languages, layers and models.
• Our analysis validates several empirical results and long-standing intuitions, and unveils new observations regarding how representations evolve in a multilingual translation model.