Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

Understand

Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters.

• The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works.
• Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs.
• In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development.