Understand
Transformers have shown great promise as an approach to Neural Machine Translation (NMT) for low-resource languages.
- However, at the same time, transformer models remain difficult to optimize and require careful tuning of hyper-parameters to be useful in this setting.
- Many NMT toolkits come with a set of default hyper-parameters, which researchers and practitioners often adopt for the sake of convenience and avoiding tuning.
- These configurations, however, have been optimized for large-scale machine translation data sets with several millions of parallel sentences for European languages like English and French.
Built on
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Statistical translation with scarce resources: a south african case study
Kato Ronald and Etienne Barnard · 2007
Earlier work this paper cites.
Processing parallel text corpora for three south african language pairs in the autshumato project
J. Hendrik Groenewald and Liza du Plooy · 2010
Earlier work this paper cites.
Dropout: A simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Exploring unsupervised word segmentation for machine translation in the south african context
Daniel R. van Niekerk · 2014
Earlier work this paper cites.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein · 2016
Earlier work this paper cites.
Similar
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Towards neural machine translation for african languages
Jade Z Abbott and Laura Martinus · 2018
Cited alongside, same era.
Training tips for the transformer model
Martin Popel and Ondřej Bojar · 2018
Cited alongside, same era.
Critical initialisation for deep signal propagation in noisy rectifier neural networks, 2018
Arnu Pretorius, Elan Van Biljon, Steve Kroon, and Herman Kamper · 2018
Cited alongside, same era.
Reducing transformer depth on demand with structured dropout
Angela Fan, Edouard Grave, and Armand Joulin · 2019
Cited alongside, same era.
Then
A focus on neural machine translation for african languages
Laura Martinus and Jade Z Abbott · 2019
Later among the works it cites.
Auto-sizing the transformer network: Improving speed, efficiency, and performance for low-resource machine translation
Kenton Murray, Jeffery Kinnison, Toan Q Nguyen, Walter Scheirer, and David Chiang · 2019
Later among the works it cites.
Facebook fair’s wmt19 news translation task submission, 2019
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov · 2019
Later among the works it cites.
Transformers without tears: Improving the normalization of self-attention
Toan Q Nguyen and Julian Salazar · 2019
Later among the works it cites.
Rezero is all you need: Fast convergence at large depth
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley · 2020
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…