The Grammar-Learning Trajectories of Neural Language Models

Understand

The learning trajectories of linguistic phenomena in humans provide insight into linguistic representation, beyond what can be gleaned from inspecting the behavior of an adult speaker.

• To apply a similar approach to analyze neural language models (NLM), it is first necessary to establish that different models are similar enough in the generalizations they make.
• In this paper, we show that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance.
• These findings suggest that there is some mutual inductive bias that underlies these models' learning of linguistic phenomena.