Can language models handle recursively nested grammatical structures? A case study on comparing models and humans

Understand

How should we compare the capabilities of language models (LMs) and humans? I draw inspiration from comparative psychology to highlight some challenges.

• In particular, I consider a case study: processing of recursively nested grammatical structures.
• Prior work suggests that LMs cannot handle these structures as reliably as humans can.
• However, the humans were provided with instructions and training, while the LMs were evaluated zero-shot.