2021

Frequency-Aware Contrastive Learning for Neural Machine Translation

Zhang, Tong, Ye, Wei, Yang, Baosong et al.

Understand

Low-frequency word prediction remains a challenge in modern neural machine translation (NMT) systems.

Recent adaptive training methods promote the output of infrequent words by emphasizing their weights in the overall training objectives.
Despite the improved recall of low-frequency words, their prediction precision is unexpectedly hindered by the adaptive objectives.
Inspired by the observation that low-frequency words form a more compact embedding space, we tackle this challenge from a representation learning perspective.

Built on

Human Behavior and The Principle of Least Effort
Zipf, G. 1949 · 1949
Earlier work this paper cites.
Statistical Significance Tests for Machine Translation Evaluation
Koehn, P. 2004 · 2004
Earlier work this paper cites.
ROUGE: A Package for Automatic Evaluation of Summaries
Lin, C.-Y. 2004 · 2004
Earlier work this paper cites.
CERT: Contrastive Self-Supervised Learning for Language Understanding
Original
Fang, H.; Wang, S.; Zhou, M.; Ding, J.; and Xie, P. 2020 · 2005
Earlier work this paper cites.
Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR)
Covington, M. A.; and McFall, J. D. 2010 · 2010
Earlier work this paper cites.
MTLD, vocd-D, and HD-D: A Validation Study of Sophisticated Approaches to Lexical Diversity Assessment
McCarthy, P. M.; and Jarvis, S. 2010 · 2010
Earlier work this paper cites.
Sequence to Sequence Learning with Neural Networks
Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014 · 2014
Earlier work this paper cites.
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D.; Cho, K.; and Bengio, Y. 2015 · 2015
Earlier work this paper cites.
Addressing the Rare Word Problem in Neural Machine Translation
Luong, M.-T.; Sutskever, I.; Le, Q.; Vinyals, O.; and Zaremba, W. 2015 · 2015
Earlier work this paper cites.
Incorporating Discrete Translation Lexicons into Neural Machine Translation
Arthur, P.; Neubig, G.; and Nakamura, S. 2016 · 2016
Earlier work this paper cites.
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
Original
Luong, M.-T.; and Manning, C. D. 2016 · 2016
Earlier work this paper cites.
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R.; Haddow, B.; and Birch, A. 2016 · 2016
Earlier work this paper cites.
Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation
Original
Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; et al. 2016 · 2016
Earlier work this paper cites.

Similar

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Hakan, I.; Khashayar, K.; and Richard, S. 2017 · 2017
Cited alongside, same era.
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Lee, J.; Cho, K.; and Hofmann, T. 2017 · 2017
Cited alongside, same era.
Focal Loss for Dense Object Detection
Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017 · 2017
Cited alongside, same era.
Attention is All You Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Cited alongside, same era.
Thumt: An open source toolkit for neural machine translation
Original
Zhang, J.; Ding, Y.; Shen, S.; Cheng, Y.; Sun, M.; Luan, H.; and Liu, Y. 2017 · 2017
Cited alongside, same era.
An Efficient Framework for Learning Sentence Representations
Logeswaran, L.; and Lee, H. 2018 · 2018
Cited alongside, same era.
Representation Degeneration Problem in Training Natural Language Generation Models
Gao, J.; He, D.; Tan, X.; Qin, T.; Wang, L.; and Liu, T.-Y. 2019 · 2019
Cited alongside, same era.
Learning Deep Representations by Mutual Information Estimation and Maximization
Hjelm, R. D.; Fedorov, A.; Lavoie-Marchildon, S.; Grewal, K.; Trischler, A.; and Bengio, Y. 2019 · 2019
Cited alongside, same era.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Jiang, S.; Ren, P.; Monz, C.; and de Rijke, M. 2019 · 2019
Cited alongside, same era.
Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
Vanmassenhove, E.; Shterionov, D.; and Way, A. 2019 · 2019
Cited alongside, same era.
Reducing Word Omission Errors in Neural Machine Translation: A Contrastive Learning Approach
Yang, Z.; Cheng, Y.; Liu, Y.; and Sun, M. 2019 · 2019
Cited alongside, same era.
A Simple Framework for Contrastive Learning of Visual Representations
Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. E. 2020 · 2020
Cited alongside, same era.

Then

Finding the Optimal Vocabulary Size for Neural Machine Translation
Gowda, T.; and May, J. 2020 · 2020
Later among the works it cites.
Token-Level Adaptive Training for Neural Machine Translation
Gu, S.; Zhang, J.; Meng, F.; Feng, Y.; Xie, W.; Zhou, J.; and Yu, D. 2020 · 2020
Later among the works it cites.
Supervised Contrastive Learning
Khosla, P.; Teterwak, P.; Wang, C.; Sarna, A.; Tian, Y.; Isola, P.; Maschinot, A.; Liu, C.; and Krishnan, D. 2020 · 2020
Later among the works it cites.
Self-Paced Learning for Neural Machine Translation
Wan, Y.; Yang, B.; Wong, D. F.; Zhou, Y.; Chao, L. S.; Zhang, H.; and Chen, B. 2020 · 2020
Later among the works it cites.
Improving Neural Language Generation with Spectrum Control
Wang, L.; Huang, J.; Huang, K.; Hu, Z.; Wang, G.; and Gu, Q. 2020 · 2020
Later among the works it cites.
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
Wang, T.; and Isola, P. 2020 · 2020
Later among the works it cites.
Uncertainty-aware curriculum learning for neural machine translation
Zhou, Y.; Yang, B.; Wong, D. F.; Wan, Y.; and Chao, L. S. 2020 · 2020
Later among the works it cites.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Original
Gao, T.; Yao, X.; and Chen, D. 2021 · 2021
Closest in time.
Contrastive Learning with Adversarial Perturbations for Conditional Text Generation
Lee, S.; Lee, D. B.; and Hwang, S. J. 2021 · 2021
Closest in time.
Towards User-Driven Neural Machine Translation
Lin, H.; Yao, L.; Yang, B.; Liu, D.; Zhang, H.; Luo, W.; Huang, D.; and Su, J. 2021 · 2021
Closest in time.
Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation
Liu, X.; Yang, B.; Liu, D.; Zhang, H.; Luo, W.; Zhang, M.; Zhang, H.; and Su, J. 2021 · 2021
Closest in time.
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
Pan, X.; Wang, M.; Wu, L.; and Li, L. 2021 · 2021
Closest in time.
Point, Disambiguate and Copy: Incorporating Bilingual Dictionaries for Neural Machine Translation
Zhang, T.; Zhang, L.; Ye, W.; Li, B.; Sun, J.; Zhu, X.; Zhao, W.; and Zhang, S. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…