Does BERT agree? Evaluating knowledge of structure dependence through agreement relations

Understand

Learning representations that accurately model semantics is an important goal of natural language processing research.

• Many semantic phenomena depend on syntactic structure.
• Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs.
• We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages.