Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding

Understand

Attention-based models have shown significant improvement over traditional algorithms in several NLP tasks.

• The Transformer, for instance, is an illustrative example that generates abstract representations of tokens inputted to an encoder based on their relationships to all tokens in a sequence.
• Recent studies have shown that although such models are capable of learning syntactic features purely by seeing examples, explicitly feeding this information to deep learning models can significantly enhance their performance.
• Leveraging syntactic information like part of speech (POS) may be particularly beneficial in limited training data settings for complex models such as the Transformer.