What if This Modified That? Syntactic Interventions via Counterfactual Embeddings

Understand

Neural language models exhibit impressive performance on a variety of tasks, but their internal reasoning may be difficult to understand.

• Prior art aims to uncover meaningful properties within model representations via probes, but it is unclear how faithfully such probes portray information that the models actually use.
• To overcome such limitations, we propose a technique, inspired by causal analysis, for generating counterfactual embeddings within models.
• In experiments testing our technique, we produce evidence that suggests some BERT-based models use a tree-distance-like representation of syntax in downstream prediction tasks.