2020

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Wang, Zirui, Lipton, Zachary C., Tsvetkov, Yulia

Understand

Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages.

However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference.
In this paper, we present the first systematic study of negative interference.
We show that, contrary to previous belief, negative interference also impacts low-resource languages.

Built on

Cross-lingual language model pretraining
Original
Guillaume Lample and Alexis Conneau. 2019 · 1901
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Original
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 1902
Earlier work this paper cites.
Multilingual neural machine translation with knowledge distillation
Original
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019 · 1902
Earlier work this paper cites.
Massively multilingual neural machine translation
Original
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019 · 1903
Earlier work this paper cites.
Beto, bentz, becas: The surprising cross-lingual effectiveness of bert
Original
Shijie Wu and Mark Dredze. 2019 · 1904
Earlier work this paper cites.
How multilingual is multilingual bert?
Original
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019 · 1906
Earlier work this paper cites.
Massively multilingual neural machine translation in the wild: Findings and challenges
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019 · 1907
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Simple, scalable adaptation for neural machine translation
Original
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. 2019 · 1909
Earlier work this paper cites.
Meta-learning with warped gradient descent
Original
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Hujun Yin, and Raia Hadsell. 2019 · 1909
Earlier work this paper cites.
Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks
Original
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019 · 1909
Earlier work this paper cites.
Investigating multilingual nmt representations at scale
Original
Sneha Reddy Kudugunta, Ankur Bapna, Isaac Caswell, Naveen Arivazhagan, and Orhan Firat. 2019 · 1909
Earlier work this paper cites.
On the cross-lingual transferability of monolingual representations
Original
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019 · 1910
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Original
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911
Earlier work this paper cites.
Emerging cross-lingual structure in pretrained language models
Original
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911
Earlier work this paper cites.
Cross-lingual name tagging and linking for 282 languages
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017 · 1958
Earlier work this paper cites.

Similar

Gradient surgery for multi-task learning
Original
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020 · 2001
Cited alongside, same era.
Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages
Original
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020 · 2003
Cited alongside, same era.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization
Original
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2003
Cited alongside, same era.
Balancing training for multilingual neural machine translation
Original
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020a · 2004
Cited alongside, same era.
To transfer or not to transfer
Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. 2005 · 2005
Cited alongside, same era.
An overview of bilevel optimization
Benoît Colson, Patrice Marcotte, and Gilles Savard. 2007 · 2007
Cited alongside, same era.
A survey on transfer learning
Sinno Jialin Pan and Qiang Yang. 2010 · 2010
Cited alongside, same era.
On handling negative transfer and imbalanced distributions in multiple source transfer learning
Liang Ge, Jing Gao, Hung Ngo, Kang Li, and Aidong Zhang. 2014 · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba. 2014 · 2014
Cited alongside, same era.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017 · 2017
Cited alongside, same era.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017 · 2017
Cited alongside, same era.

Then

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Later among the works it cites.
Online learning rate adaptation with hypergradient descent
Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. 2018 · 2018
Later among the works it cites.
Xnli: Evaluating cross-lingual sentence representations
Original
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Later among the works it cites.
Meta-learning for low-resource neural machine translation
Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018 · 2018
Later among the works it cites.
Universal dependencies 2.2
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018 · 2018
Later among the works it cites.
Know what you don’t know: Unanswerable questions for squad
Original
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 · 2018
Later among the works it cites.
On the limitations of unsupervised bilingual dictionary induction
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018 · 2018
Later among the works it cites.
Towards more reliable transfer learning
Zirui Wang and Jaime Carbonell. 2018 · 2018
Later among the works it cites.
Similarity of neural network representations revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019 · 2019
Later among the works it cites.
Polyglot contextual representations improve crosslingual transfer
Phoebe Mulcaire, Jungo Kasai, and Noah A Smith. 2019 · 2019
Later among the works it cites.
Characterizing and avoiding negative transfer
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. 2019 · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 · 2019
Later among the works it cites.
Multilingual alignment of contextual word representations
Steven Cao, Nikita Kitaev, and Dan Klein. 2020 · 2020
Closest in time.
Cross-lingual ability of multilingual bert: An empirical study
K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…