Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Understand

Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed.

• We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations.
• In this work we aim to alleviate these issues by proposing $\mathcal{M}\mathcal{F}_\beta$, a decomposable metric that builds on two pillars.
• The first is the principle of meaning preservation $\mathcal{M}$: it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (fine-grained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR.