2022

Using natural language prompts for machine translation

Garcia, Xavier, Firat, Orhan

Understand

We explore the use of natural language prompts for controlling various aspects of the outputs generated by machine translation models.

We demonstrate that natural language prompts allow us to influence properties like formality or specific dialect of the output.
We show that using language names to control the output language of multilingual translation models enables positive transfer for unseen language pairs.
This unlocks the ability to translate into languages not seen during fine-tuning by using their English names.

Built on

The missing ingredient in zero-shot neural machine translation
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a · 1903
Earlier work this paper cites.
Mass: Masked sequence to sequence pre-training for language generation
Original
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019 · 1905
Earlier work this paper cites.
Tagged back-translation
Original
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019 · 1906
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
Unsupervised cross-lingual representation learning at scale
Original
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1911
Earlier work this paper cites.
Microsoft research asia’s systems for wmt19
Original
Yingce Xia, Xu Tan, Fei Tian, Fei Gao, Weicong Chen, Yang Fan, Linyuan Gong, Yichong Leng, Renqian Luo, Yiren Wang, et al. 2019 · 1911
Earlier work this paper cites.
The pronouns of power and solidarity
Roger Brown and Albert Gilman. 1960 · 1960
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Statistical significance tests for machine translation evaluation
Philipp Koehn. 2004 · 2004
Earlier work this paper cites.
Reference language based unsupervised neural machine translation
Original
Zuchao Li, Hai Zhao, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2020 · 2004
Earlier work this paper cites.
Cross-lingual supervision improves unsupervised neural machine translation
Original
Mingxuan Wang, Hongxiao Bai, Hai Zhao, and Lei Li. 2020 · 2004
Earlier work this paper cites.
Improving massively multilingual neural machine translation and zero-shot translation
Original
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020 · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 2005
Earlier work this paper cites.
Multilingual translation with extensible multilingual pretraining and finetuning
Original
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020 · 2008
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.

Similar

Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014
Cited alongside, same era.
Multi-way, multilingual neural machine translation with a shared attention mechanism
Original
Orhan Firat, KyungHyun Cho, and Yoshua Bengio. 2016 · 2016
Cited alongside, same era.
Ask me anything: Dynamic memory networks for natural language processing
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016 · 2016
Cited alongside, same era.
Google’s multilingual neural machine translation system: Enabling zero-shot translation
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017 · 2017
Cited alongside, same era.
Multilingual neural machine translation with task-specific attention
Original
Graeme Blackwood, Miguel Ballesteros, and Todd Ward. 2018 · 2018
Cited alongside, same era.
Jax: composable transformations of python+numpy programs
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. 2018 · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Understanding back-translation at scale
Original
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 · 2018
Cited alongside, same era.
A call for clarity in reporting bleu scores
Original
Matt Post. 2018 · 2018
Cited alongside, same era.
When and why are pre-trained word embeddings useful for neural machine translation?
Original
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. 2018 · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Cited alongside, same era.
Zero-shot dual machine translation
Original
Lierni Sestorain, Massimiliano Ciaramita, Christian Buck, and Thomas Hofmann. 2018 · 2018
Cited alongside, same era.

Then

Nict’s unsupervised neural and statistical machine translation systems for the wmt19 news translation task
Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019 · 2019
Later among the works it cites.
Multi-agent dual learning
Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019 · 2019
Later among the works it cites.
A multilingual view of unsupervised machine translation
Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur Parikh. 2020 · 2020
Later among the works it cites.
Multilingual denoising pre-training for neural machine translation
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020 · 2020
Later among the works it cites.
Muppet: Massive multi-task representations with pre-finetuning
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021 · 2021
Later among the works it cites.
Ext5: Towards extreme multi-task scaling for transfer learning
Original
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. 2021 · 2021
Later among the works it cites.
Towards making the most of multilingual pretraining for zero-shot neural machine translation
Original
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2021 · 2021
Later among the works it cites.
Harnessing multilinguality in unsupervised machine translation for rare languages
Xavier Garcia, Aditya Siddhant, Orhan Firat, and Ankur Parikh. 2021 · 2021
Later among the works it cites.
Few-shot learning with multilingual language models
Original
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021 · 2021
Later among the works it cites.
Samanantar: The largest publicly available parallel corpora collection for 11 indic languages
Original
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Divyanshu Kakwani, Navneet Kumar, et al. 2021 · 2021
Later among the works it cites.
Multitask prompted training enables zero-shot task generalization
Original
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021 · 2021
Later among the works it cites.
Finetuned language models are zero-shot learners
Original
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021 · 2021
Later among the works it cites.
Language tags matter for zero-shot neural machine translation
Original
Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021 · 2021
Later among the works it cites.
mT5: A massively multilingual pre-trained text-to-text transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…