BLiMP: The Benchmark of Linguistic Minimal Pairs for English

Understand

We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English.

• BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics.
• The data is automatically generated according to expert-crafted grammars, and aggregate human agreement with the labels is 96.4%.
• We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs.