2019

Regularized Context Gates on Transformer for Machine Translation

Li, Xintong, Liu, Lemao, Wang, Rui et al.

Understand

Context gates are effective to control the contributions from the source and target contexts in the recurrent neural network (RNN) based neural machine translation (NMT).

However, it is challenging to extend them into the advanced Transformer architecture, which is more complicated than RNN.
This paper first provides a method to identify source and target contexts and then introduce a gate mechanism to control the source and target contributions in Transformer.
In addition, to further reduce the bias problem in the gate mechanism, this paper proposes a regularization method to guide the learning of the gates with supervision automatically generated using pointwise mutual information.

Built on

Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Statistical phrase-based translation
Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003 · 2003
Earlier work this paper cites.
A hierarchical phrase-based model for statistical machine translation
David Chiang. 2005 · 2005
Earlier work this paper cites.
Normalized (pointwise) mutual information in collocation extraction
Gerlof Bouma. 2009 · 2009
Earlier work this paper cites.
Statistical machine translation
Philipp Koehn. 2009 · 2009
Earlier work this paper cites.
Collocation extraction using monolingual word alignment method
Zhanyi Liu, Haifeng Wang, Hua Wu, and Sheng Li. 2009 · 2009
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Original
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015 · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015 · 2015
Earlier work this paper cites.

Similar

Highway networks
Original
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015 · 2015
Cited alongside, same era.
Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016 · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Cited alongside, same era.
Neural machine translation with supervised attention
Original
Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016 · 2016
Cited alongside, same era.
Supervised attentions for neural machine translation
Original
Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016 · 2016
Cited alongside, same era.
Convolutional sequence to sequence learning
Original
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017 · 2017
Cited alongside, same era.
Context gates for neural machine translation
Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Thumt: an open source toolkit for neural machine translation
Original
Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. 2017 · 2017
Cited alongside, same era.
Syntax-directed attention for neural machine translation
Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2018 · 2018
Cited alongside, same era.

Then

Towards robust neural machine translation
Original
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018 · 2018
Later among the works it cites.
Pervasive attention: 2d convolutional neural networks for sequence-to-sequence prediction
Original
Maha Elbayad, Laurent Besacier, and Jakob Verbeek. 2018 · 2018
Later among the works it cites.
Layer-wise coordination between encoder and decoder for neural machine translation
Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018 · 2018
Later among the works it cites.
Target foresight based attention for neural machine translation
Xintong Li, Lemao Liu, Zhaopeng Tu, Shuming Shi, and Max Meng. 2018 · 2018
Later among the works it cites.
Bag-of-words as target for neural machine translation
Original
Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018 · 2018
Later among the works it cites.
Improving the transformer translation model with document-level context
Original
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018 · 2018
Later among the works it cites.
Addressing troublesome words in neural machine translation
Yang Zhao, Jiajun Zhang, Zhongjun He, Chengqing Zong, and Hua Wu. 2018 · 2018
Later among the works it cites.
On the word alignment from neural machine translation
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019 · 2019
Closest in time.
Neural machine translation with target-attention model
Mingming Yang, Min Zhang, Kehai Chen, Rui Wang, and Tiejun Zhao. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…