CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge

Understand

Frontier large language models (LLMs) are developed by researchers and practitioners with skewed cultural backgrounds and on datasets with skewed sources.

• However, LLMs' (lack of) multicultural knowledge cannot be effectively assessed with current methods for developing benchmarks.
• Existing multicultural evaluations primarily rely on expensive and restricted human annotations or potentially outdated internet resources.
• Thus, they struggle to capture the intricacy, dynamics, and diversity of cultural norms.