Analyzing Compositionality-Sensitivity of NLI Models

Understand

Success in natural language inference (NLI) should require a model to understand both lexical and compositional semantics.

• However, through adversarial evaluation, we find that several state-of-the-art models with diverse architectures are over-relying on the former and fail to use the latter.
• Further, this compositionality unawareness is not reflected via standard evaluation on current datasets.
• We show that removing RNNs in existing models or shuffling input words during training does not induce large performance loss despite the explicit removal of compositional information.