Can neural networks acquire a structural bias from raw linguistic data?

Understand

We evaluate whether BERT, a widely used neural network for sentence processing, acquires an inductive bias towards forming structural generalizations through pretraining on raw data.

• We conduct four experiments testing its preference for structural vs.
• linear generalizations in different structure-dependent phenomena.
• We find that BERT makes a structural generalization in 3 out of 4 empirical domains---subject-auxiliary inversion, reflexive binding, and verb tense detection in embedded clauses---but makes a linear generalization when tested on NPI licensing.