2017

Neural Machine Translation Training in a Multi-Domain Scenario

Sajjad, Hassan, Durrani, Nadir, Dalvi, Fahim et al.

Understand

In this paper, we explore alternative ways to train a neural machine translation system in a multi-domain scenario.

We investigate data concatenation (with fine tuning), model stacking (multi-level fine tuning), data selection and multi-model ensemble.
Our findings show that the best translation quality can be achieved by building an initial system on a concatenation of available out-of-domain data and then fine-tuning it on in-domain data.
Model stacking works best when training begins with the furthest out-of-domain data and the model is incrementally fine-tuned with the next furthest domain and so on.

Built on

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the Association for Computational Linguistics (ACL’02) , Philadelphia, PA, USA, 2002
2002
Earlier work this paper cites.
A. Fujii, M. Utiyama, M. Yamamoto, and T. Utsuro, “Overview of the patent translation task at the ntcir-8 workshop,” in In Proceedings of the 8th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Cross-lingual Information Access , 2010, pp. 293–302
2010
Earlier work this paper cites.
A. Axelrod, X. He, and J. Gao, “Domain adaptation via pseudo in-domain data selection,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing , ser. EMNLP ’11, Edinburgh, United Kingdom, 2011
2011
Earlier work this paper cites.
F. Guzmán, H. Sajjad, S. Vogel, and A. Abdelali, “The AMARA corpus: Building resources for translating the web’s educational content,” in Proceedings of the 10th International Workshop on Spoken Language Technology (IWSLT-13) , December 2013
2013
Earlier work this paper cites.
H. Sajjad, F. Guzmán, P. Nakov, A. Abdelali, K. Murray, F. A. Obaidli, and S. Vogel, “QCRI at IWSLT 2013: Experiments in Arabic-English and English-Arabic spoken language translation,” in Proceedings of the 10th International Workshop on Spoken Language Technology (IWSLT-13) , December 2013
2013
Earlier work this paper cites.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico, “Report on the 11th IWSLT Evaluation Campaign,” Proceedings of the International Workshop on Spoken Language Translation, Lake Tahoe, US , 2014
2014
Earlier work this paper cites.
O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna, “Findings of the 2014 workshop on statistical machine translation,” in Proceedings of the Ninth Workshop on Statistical Machine Translation , Baltimore, Maryland, USA, June 2014
2014
Earlier work this paper cites.
M.-T. Luong and C. D. Manning, “Stanford Neural Machine Translation Systems for Spoken Language Domains,” in Proceedings of the International Workshop on Spoken Language Translation , Da Nang, Vietnam, December 2015
2015
Earlier work this paper cites.
N. Durrani, H. Sajjad, S. Joty, A. Abdelali, and S. Vogel, “Using Joint Models for Domain Adaptation in Statistical Machine Translation,” in Proceedings of the Fifteenth Machine Translation Summit (MT Summit XV) . Florida, USA: AMTA, To Appear 2015
2015
Earlier work this paper cites.
S. Joty, H. Sajjad, N. Durrani, K. Al-Mannai, A. Abdelali, and S. Vogel, “How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , Lisbon, Portugal, September 2015
2015
Earlier work this paper cites.

Similar

D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR , 2015. [Online]. Available: http://arxiv.org/pdf/1409.0473v6.pdf
Original
2015
Cited alongside, same era.
M. Freitag and Y. Al-Onaizan, “Fast domain adaptation for neural machine translation,” CoRR , vol. abs/1612.06897, 2016. [Online]. Available: http://arxiv.org/abs/1612.06897
Original
2016
Cited alongside, same era.
C. Servan, J. M. Crego, and J. Senellart, “Domain specialization: a post-training domain adaptation for neural machine translation,” CoRR , vol. abs/1612.06141, 2016. [Online]. Available: http://arxiv.org/abs/1612.06141
Original
2016
Cited alongside, same era.
C. Kobus, J. M. Crego, and J. Senellart, “Domain control for neural machine translation,” CoRR , vol. abs/1612.06140, 2016. [Online]. Available: http://arxiv.org/abs/1612.06140
Original
2016
Cited alongside, same era.
P. Williams, R. Sennrich, M. Nadejde, M. Huck, B. Haddow, and O. Bojar, “Edinburgh’s statistical machine translation systems for wmt16,” in Proceedings of the First Conference on Machine Translation . Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 399–410. [Online]. Available: http://www.aclweb.org/anthology/W16-2327
2016
Cited alongside, same era.
R. Sennrich, B. Haddow, and A. Birch, “Edinburgh neural machine translation systems for wmt 16,” in Proceedings of the First Conference on Machine Translation . Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 371–376. [Online]. Available: http://www.aclweb.org/anthology/W16-2323
2016
Cited alongside, same era.
N. Durrani, H. Sajjad, S. Joty, and A. Abdelali, “A deep fusion model for domain adaptation in phrase-based mt,” in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers . Osaka, Japan: The COLING 2016 Organizing Committee, December 2016, pp. 3177–3187. [Online]. Available: http://aclweb.org/anthology/C16-1299
2016
Cited alongside, same era.
M. Cettolo, “An Arabic-Hebrew parallel corpus of TED talks,” in Proceedings of the AMTA Workshop on Semitic Machine Translation (SeMaT) , Austin, US-TX, November 2016
2016
Cited alongside, same era.
M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen, “The united nations parallel corpus v1.0,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016 , 2016
2016
Cited alongside, same era.
P. Lison and J. Tiedemann, “Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles,” in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) . European Language Resources Association (ELRA), may 2016
2016
Cited alongside, same era.

Then

R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Translation of Rare Words with Subword Units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Berlin, Germany: Association for Computational Linguistics, August 2016, pp. 1715–1725. [Online]. Available: http://www.aclweb.org/anthology/P16-1162
2016
Later among the works it cites.
A. Abdelali, K. Darwish, N. Durrani, and H. Mubarak, “Farasa: A fast and furious segmenter for arabic,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations . San Diego, California: Association for Computational Linguistics, June 2016, pp. 11–16. [Online]. Available: http://www.aclweb.org/anthology/N16-3003
2016
Later among the works it cites.
C. Chu, R. Dabre, and S. Kurohashi, “An empirical comparison of simple domain adaptation methods for neural machine translation,” CoRR , vol. abs/1701.03214, 2017. [Online]. Available: http://arxiv.org/abs/1701.03214
Original
2017
Closest in time.
M. van der Wees, A. Bisazza, and C. Monz, “Dynamic data selection for neural machine translation,” in Proceedings of the the Conference on Empirical Methods in Natural Language Processing , September 2017
2017
Closest in time.
R. Wang, M. Utiyama, L. Liu, K. Chen, and E. Sumita, “Instance weighting for neural machine translation domain adaptation,” in Proceedings of the the Conference on Empirical Methods in Natural Language Processing, , September 2017
2017
Closest in time.
B. Chen, C. Cherry, G. Foster, and S. Larkin, “Cost weighting for neural machine translation domain adaptation,” in Proceedings of the First Workshop on Neural Machine Translation , September 2017
2017
Closest in time.
R. Wang, A. Finch, M. Utiyama, and E. Sumita, “Sentence embedding for neural machine translation domain adaptation,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers) , August 2017
2017
Closest in time.
H. Sajjad, F. Dalvi, N. Durrani, A. Abdelali, Y. Belinkov, and S. Vogel, “Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging,” in Proceedings of the Association for Computational Linguistics , 2017
2017
Closest in time.
R. Sennrich, O. Firat, K. Cho, A. Birch, B. Haddow, J. Hitschler, M. Junczys-Dowmunt, S. L”aubli, A. V. Miceli Barone, J. Mokry, and M. Nadejde, “Nematus: a toolkit for neural machine translation,” in Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics . Valencia, Spain: Association for Computational Linguistics, April 2017, pp. 65–68. [Online]. Available: http://aclweb.org/anthology/E17-3017
2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…