Attention Can Reflect Syntactic Structure (If You Let It)

Understand

Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism.

• However, much of such work focused almost exclusively on English -- a language with rigid word order and a lack of inflectional morphology.
• In this study, we present decoding experiments for multilingual BERT across 18 languages in order to test the generalizability of the claim that dependency syntax is reflected in attention patterns.
• We show that full trees can be decoded above baseline accuracy from single attention heads, and that individual relations are often tracked by the same heads across languages.