Targeted Syntactic Evaluation of Language Models

Understand

We present a dataset for evaluating the grammaticality of the predictions of a language model.

• We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.
• The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items.
• We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one.