LLMs' Understanding of Natural Language Revealed

Understand

Large language models (LLMs) are the result of a massive experiment in bottom-up, data-driven reverse engineering of language at scale.

• Despite their utility in a number of downstream NLP tasks, ample research has shown that LLMs are incapable of performing reasoning in tasks that require quantification over and the manipulation of symbolic variables (e.g., planning and problem solving); see for example [25][26].
• In this document, however, we will focus on testing LLMs for their language understanding capabilities, their supposed forte.
• As we will show here, the language understanding capabilities of LLMs have been widely exaggerated.