Inducing Syntactic Trees from BERT Representations

Understand

We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words.

• Our hypothesis is that removing a reducible word (e.g.
• an adjective) does not affect the representation of other words so much as removing e.g.
• the main verb, which makes the sentence ungrammatical and of "high surprise" for the language model.