GLEE: A Unified Framework and Benchmark for Language-based Economic Environments

Understand

Large Language Models (LLMs) show significant potential in economic and strategic interactions, where communication via natural language is often prevalent.

• This raises key questions: Do LLMs behave rationally? How do they perform compared to humans? Do they tend to reach an efficient and fair outcome? What is the role of natural language in strategic interaction? How do characteristics of the economic environment influence these dynamics? These questions become crucial concerning the economic and societal implications of integrating LLM-based agents into real-world data-driven systems, such as online retail platforms and recommender systems.
• To answer these questions, we introduce a benchmark for standardizing research on two-player, sequential, language-based games.
• Inspired by the economic literature, we define three base families of games with consistent parameterization, degrees of freedom and economic measures to evaluate agents' performance (self-gain), as well as the game outcome (efficiency and fairness).