The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages

Understand

Instruction tuned large language models (LLMs), such as ChatGPT, demonstrate remarkable performance in a wide range of tasks.

• Despite numerous recent studies that examine the performance of instruction-tuned LLMs on various NLP benchmarks, there remains a lack of comprehensive investigation into their ability to understand cross-lingual sociopragmatic meaning (SM), i.e., meaning embedded within social and interactive contexts.
• This deficiency arises partly from SM not being adequately represented in any of the existing benchmarks.
• To address this gap, we present SPARROW, an extensive multilingual benchmark specifically designed for SM understanding.