2019

Character-based NMT with Transformer

Gupta, Rohit, Besacier, Laurent, Dymetman, Marc et al.

Understand

Character-based translation has several appealing advantages, but its performance is in general worse than a carefully tuned BPE baseline.

In this paper we study the impact of character-based input and output with the Transformer architecture.
In particular, our experiments on EN-DE show that character-based Transformer models are more robust than their BPE counterpart, both when translating noisy text, and when translating text from a different domain.
To obtain comparable BLEU scores in clean, in-domain data and close the gap with BPE-based models we use known techniques to train deeper Transformer models.

Built on

V. Karpukhin, O. Levy, J. Eisenstein, and M. Ghazvininejad, “Training on synthetic noise improves robustness to natural noise in machine translation,” CoRR , vol. abs/1902.01509, 2019. [Online]. Available: http://arxiv.org/abs/1902.01509
Original
1902
Earlier work this paper cites.
Vaibhav, S. Singh, C. Stewart, and G. Neubig, “Improving robustness of machine translation with synthetic noise,” CoRR , vol. abs/1902.09508, 2019. [Online]. Available: http://arxiv.org/abs/1902.09508
Original
1902
Earlier work this paper cites.
S. Ding, A. Renduchintala, and K. Duh, “A call for prudent choice of subword merge operations,” CoRR , vol. abs/1905.10453, 2019. [Online]. Available: http://arxiv.org/abs/1905.10453
Original
1905
Earlier work this paper cites.
Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” CoRR , vol. abs/1906.01787, 2019. [Online]. Available: http://arxiv.org/abs/1906.01787
Original
1906
Earlier work this paper cites.
H. Ney, U. Essen, and R. Kneser, “On structuring probabilistic dependencies in stochastic language modelling,” Computer Speech and Language , vol. 8, pp. 1–38, 1994
1994
Earlier work this paper cites.
P. Koehn, “Europarl: A parallel corpus for statistical machine translation,” vol. 5, 11 2004
2004
Earlier work this paper cites.
D. Vilar, J.-T. Peter, and H. Ney, “Can we translate letters?” in Proceedings of the Second Workshop on Statistical Machine Translation , ser. StatMT ’07. Stroudsburg, PA, USA: Association for Computational Linguistics, 2007, pp. 33–39. [Online]. Available: http://dl.acm.org/citation.cfm?id=1626355.1626360
2007
Earlier work this paper cites.
J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset Shift in Machine Learning . The MIT Press, 2009
2009
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, Y. W. Teh and M. Titterington, Eds., vol. 9. Chia Laguna Resort, Sardinia, Italy: PMLR, 13–15 May 2010, pp. 249–256. [Online]. Available: http://proceedings.mlr.press/v9/glorot10a.html
2010
Earlier work this paper cites.
V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th International Conference on International Conference on Machine Learning , ser. ICML’10. USA: Omnipress, 2010, pp. 807–814. [Online]. Available: http://dl.acm.org/citation.cfm?id=3104322.3104425
2010
Earlier work this paper cites.
M. Li, Y. Zhao, D. Zhang, and M. Zhou, “Adaptive development data selection for log-linear model in statistical machine translation,” in Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010) . Beijing, China: Coling 2010 Organizing Committee, Aug. 2010, pp. 662–670. [Online]. Available: https://www.aclweb.org/anthology/C10-1075
2010
Earlier work this paper cites.
T. Mikolov, I. Sutskever, A. Deoras, H.-S. Le, S. Kombrink, and J. Cernocký, “Subword language modeling with neural networks,” 2011
2011
Earlier work this paper cites.
A. Axelrod, X. He, and J. Gao, “Domain adaptation via pseudo in-domain data selection,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing , ser. EMNLP ’11. Stroudsburg, PA, USA: Association for Computational Linguistics, 2011, pp. 355–362. [Online]. Available: http://dl.acm.org/citation.cfm?id=2145432.2145474
2011
Earlier work this paper cites.
T. Zesch, “Measuring contextual fitness using error contexts extracted from the Wikipedia revision history,” in Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics . Avignon, France: Association for Computational Linguistics, Apr. 2012, pp. 529–538. [Online]. Available: https://www.aclweb.org/anthology/E12-1054
2012
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems 26 , C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 3111–3119
2013
Earlier work this paper cites.
G. Neubig, T. Watanabe, S. Mori, and T. Kawahara, “Substring-based machine translation,” Machine Translation , vol. 27, no. 2, pp. 139–166, June 2013. [Online]. Available: http://dx.doi.org/10.1007/s10590-013-9136-6
2013
Earlier work this paper cites.
J. R. Smith, H. Saint-Amand, M. Plamada, P. Koehn, C. Callison-Burch, and A. Lopez, “Dirt cheap web-scale parallel text from the common crawl,” vol. 1, 07 2013
2013
Earlier work this paper cites.
K. Wisniewski, K. Schöne, L. Nicolas, C. Vettori, A. Boyd, D. Meurers, A. Abel, and J. Hana, “Merlin: An online trilingual learner corpus empirically grounding the european reference levels in authentic learner data,” 10 2013
2013
Earlier work this paper cites.

Similar

J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in In EMNLP , 2014
2014
Cited alongside, same era.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in International Conference on Learning Representations , 2014. [Online]. Available: http://arxiv.org/abs/1312.6199
Original
2014
Cited alongside, same era.
M. Federico, S. Stücker, and F. Yvon, “International workshop onspoken language translation,” in Proceedings of theInternational Workshop onSpoken Language Translation . Lake Tahoe, CA, USA: Association for Computational Linguistics, Dec. 2014
2014
Cited alongside, same era.
P. Wang, Y. Qian, F. K. Soong, L. He, and H. Zhao, “Part-of-speech tagging with bidirectional long short-term memory recurrent neural network,” CoRR , vol. abs/1510.06168, 2015. [Online]. Available: http://arxiv.org/abs/1510.06168
Original
2015
Cited alongside, same era.
M. Tan, B. Xiang, and B. Zhou, “Lstm-based deep learning models for non-factoid answer selection,” CoRR , vol. abs/1511.04108, 2015. [Online]. Available: http://arxiv.org/abs/1511.04108
Original
2015
Cited alongside, same era.
M. Ballesteros, C. Dyer, and N. A. Smith, “Improved transition-based parsing by modeling characters instead of words with lstms,” CoRR , vol. abs/1508.00657, 2015. [Online]. Available: http://arxiv.org/abs/1508.00657
Original
2015
Cited alongside, same era.
W. Ling, I. Trancoso, C. Dyer, and A. W. Black, “Character-based neural machine translation,” CoRR , vol. abs/1511.04586, 2015. [Online]. Available: http://arxiv.org/abs/1511.04586
Original
2015
Cited alongside, same era.
R. K. Srivastava, K. Greff, and J. Schmidhuber, “Highway networks,” CoRR , vol. abs/1505.00387, 2015. [Online]. Available: http://arxiv.org/abs/1505.00387
Original
2015
Cited alongside, same era.
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” CoRR , vol. abs/1508.07909, 2015. [Online]. Available: http://arxiv.org/abs/1508.07909
Original
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
Original
2015
Cited alongside, same era.
I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in International Conference on Learning Representations , 2015. [Online]. Available: http://arxiv.org/abs/1412.6572
Original
2015
Cited alongside, same era.
J. Bradbury, S. Merity, C. Xiong, and R. Socher, “Quasi-recurrent neural networks,” CoRR , vol. abs/1611.01576, 2016. [Online]. Available: http://arxiv.org/abs/1611.01576
Original
2016
Cited alongside, same era.

Then

W. Chen, E. Matusov, S. Khadivi, and J. Peter, “Guided alignment training for topic-aware neural machine translation,” CoRR , vol. abs/1607.01628, 2016. [Online]. Available: http://arxiv.org/abs/1607.01628
Original
2016
Later among the works it cites.
B. Heinzerling and M. Strube, “Bpemb: Tokenization-free pre-trained subword embeddings in 275 languages,” CoRR , vol. abs/1710.02187, 2017. [Online]. Available: http://arxiv.org/abs/1710.02187
Original
2017
Later among the works it cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” CoRR , vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
Original
2017
Later among the works it cites.
I. Yamada, H. Shindo, H. Takeda, and Y. Takefuji, “Learning distributed representations of texts and entities from knowledge base,” CoRR , vol. abs/1705.02494, 2017. [Online]. Available: http://arxiv.org/abs/1705.02494
Original
2017
Later among the works it cites.
A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “Supervised learning of universal sentence representations from natural language inference data,” CoRR , vol. abs/1705.02364, 2017. [Online]. Available: http://arxiv.org/abs/1705.02364
Original
2017
Later among the works it cites.
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics , vol. 5, pp. 135–146, 2017
2017
Later among the works it cites.
M. J. Denkowski and G. Neubig, “Stronger baselines for trustable results in neural machine translation,” CoRR , vol. abs/1706.09733, 2017. [Online]. Available: http://arxiv.org/abs/1706.09733
Original
2017
Later among the works it cites.
Proceedings of the 4th Workshop on Asian Translation (WAT2017) . Taipei, Taiwan: Asian Federation of Natural Language Processing, Nov. 2017. [Online]. Available: https://www.aclweb.org/anthology/W17-5700
2017
Later among the works it cites.
Y. Belinkov and Y. Bisk, “Synthetic and natural noise both break neural machine translation,” CoRR , vol. abs/1711.02173, 2017. [Online]. Available: http://arxiv.org/abs/1711.02173
Original
2017
Later among the works it cites.
P. Koehn and R. Knowles, “Six challenges for neural machine translation,” in Proceedings of the First Workshop on Neural Machine Translation . Vancouver: Association for Computational Linguistics, Aug. 2017, pp. 28–39. [Online]. Available: https://www.aclweb.org/anthology/W17-3204
2017
Later among the works it cites.
H. S. Y. B. Nadir Durrani, Fahim Dalvi and P. Nakov, “What is in a translation unit? comparing character and subword representations beyond translation,” 2018. [Online]. Available: https://openreview.net/pdf?id=B1x0E2C5tQ
2018
Later among the works it cites.
C. Cherry, G. Foster, A. Bapna, O. Firat, and W. Macherey, “Revisiting character-based neural machine translation with capacity and compression,” CoRR , vol. abs/1808.09943, 2018. [Online]. Available: http://arxiv.org/abs/1808.09943
Original
2018
Later among the works it cites.
R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones, “Character-level language modeling with deeper self-attention,” CoRR , vol. abs/1808.04444, 2018. [Online]. Available: http://arxiv.org/abs/1808.04444
Original
2018
Later among the works it cites.
T. Kenter, L. Jones, and D. Hewlett, Eds., Byte-level Machine Reading across Morphologically Varied Languages , 2018. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/download/16605/16145
2018
Later among the works it cites.
J. Kreutzer and A. Sokolov, “Learning to segment inputs for NMT favors character-level processing,” CoRR , vol. abs/1810.01480, 2018. [Online]. Available: http://arxiv.org/abs/1810.01480
Original
2018
Later among the works it cites.
A. Bapna, M. X. Chen, O. Firat, Y. Cao, and Y. Wu, “Training deeper neural machine translation models with transparent attention,” CoRR , vol. abs/1808.07561, 2018. [Online]. Available: http://arxiv.org/abs/1808.07561
Original
2018
Later among the works it cites.
A. Bérard, I. Calapodescu, and C. Roux, “Naver labs europe’s systems for the wmt19 machine translation robustness task,” arXiv preprint arXiv:1907.06488 , 2019
Original
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…