UnNatural Language Inference

Understand

Recent investigations into the inner-workings of state-of-the-art large-scale pre-trained Transformer-based Natural Language Understanding (NLU) models indicate that they appear to know humanlike syntax, at least to some extent.

• We provide novel evidence that complicates this claim: we find that state-of-the-art Natural Language Inference (NLI) models assign the same labels to permuted examples as they do to the original, i.e.
• they are largely invariant to random word-order permutations.
• This behavior notably differs from that of humans; we struggle with ungrammatical sentences.