The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions

Understand

A common language with standardized definitions is crucial for effective climate discussions.

• However, concerns exist about LLMs misrepresenting climate terms.
• We compared 300 official IPCC glossary definitions with those generated by GPT-4o-mini, Llama3.1 8B, and Mistral 7B, analyzing adherence, robustness, and readability using SBERT sentence embeddings.
• The LLMs scored an average adherence of $0.57-0.59 \pm 0.15$, and their definitions proved harder to read than the originals.