GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models

Understand

The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs).

• However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods.
• This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references.
• To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results.