Are LLM-based Evaluators Confusing NLG Quality Criteria?

Understand

Some prior work has shown that LLMs perform well in NLG evaluation for different tasks.

• However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability.
• For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves.
• So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved.