Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans

Understand

Recursive processing in sentence comprehension is considered a hallmark of human linguistic abilities.

• However, its underlying neural mechanisms remain largely unknown.
• We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing, namely the storing of grammatical number and gender information in working memory and its use in long-distance agreement (e.g., capturing the correct number agreement between subject and verb when they are separated by other phrases).
• Although the network, a recurrent architecture with Long Short-Term Memory units, was solely trained to predict the next word in a large corpus, analysis showed the emergence of a very sparse set of specialized units that successfully handled local and long-distance syntactic agreement for grammatical number.