Understand
We investigate two specific manifestations of compositionality in Neural Machine Translation (NMT) : (1) Productivity - the ability of the model to extend its predictions beyond the observed length in training data and (2) Systematicity - the ability of the model to systematically recombine known parts and rules.
- We evaluate a standard Sequence to Sequence model on tests designed to assess these two properties in NMT.
- We quantitatively demonstrate that inadequate temporal processing, in the form of poor encoder representations is a bottleneck for both Productivity and Systematicity.
- We propose a simple pre-training mechanism which alleviates model performance on the two properties and leads to a significant improvement in BLEU scores.
Built on
Report on the 11th iwslt evaluation campaign, iwslt 2014
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico · 2014
Earlier work this paper cites.
On the properties of neural machine translation: Encoder-decoder approaches
K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. V. Le · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
D. Bahdanau, K. Cho, and Y. Bengio · 2015
Earlier work this paper cites.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Earlier work this paper cites.
Similar
Effective approaches to attention-based neural machine translation
T. Luong, H. Pham, and C. D. Manning · 2015
Cited alongside, same era.
Sequence to sequence-video to text
S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko · 2015
Cited alongside, same era.
O. Vinyals and Q. Le · 2015
Cited alongside, same era.
B. M. Lake and M. Baroni · 2017
Cited alongside, same era.
Then
Neural machine translation and sequence-to-sequence models: A tutorial
G. Neubig · 2017
Later among the works it cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017
Later among the works it cites.
Jump to better conclusions: Scan both left and right
J. Bastings, M. Baroni, J. Weston, K. Cho, and D. Kiela · 2018
Later among the works it cites.
The compositionality of neural networks: integrating symbolism and connectionism
D. Hupkes, V. Dankers, M. Mul, and E. Bruni · 2019
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…