Probing for Constituency Structure in Neural Language Models

Understand

In this paper, we investigate to which extent contextual neural language models (LMs) implicitly learn syntactic structure.

• More concretely, we focus on constituent structure as represented in the Penn Treebank (PTB).
• Using standard probing techniques based on diagnostic classifiers, we assess the accuracy of representing constituents of different categories within the neuron activations of a LM such as RoBERTa.
• In order to make sure that our probe focuses on syntactic knowledge and not on implicit semantic generalizations, we also experiment on a PTB version that is obtained by randomly replacing constituents with each other while keeping syntactic structure, i.e., a semantically ill-formed but syntactically well-formed version of the PTB.