PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?

Understand

Current role-play studies often rely on unvalidated LLM-as-a-judge paradigms, which may fail to reflect how humans perceive role fidelity.

• A key prerequisite for human-aligned evaluation is role identification, the ability to recognize who is speaking based on dialogue context.
• We argue that any meaningful judgment of role-playing quality (how well a character is played) fundamentally depends on first correctly attributing words and actions to the correct persona (who is speaking).
• We present PersonaEval, the first benchmark designed to test whether LLM evaluators can reliably identify human roles.