Quantity doesn't buy quality syntax with neural language models

Understand

Recurrent neural networks can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabilities to ungrammatical words.

• We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained.
• We find that gains from increasing network size are minimal beyond a certain point.
• Likewise, expanding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance.