VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Understand

Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages.

• However, much of this work only relies on the shared vocabulary and bilingual contexts to encourage the correlation across languages, which is loose and implicit for aligning the contextual representations between languages.
• In this paper, we plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages.
• It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language.