An Evaluation on Large Language Model Outputs: Discourse and Memorization

Understand

We present an empirical evaluation of various outputs generated by nine of the most widely-available large language models (LLMs).

• Our analysis is done with off-the-shelf, readily-available tools.
• We find a correlation between percentage of memorized text, percentage of unique text, and overall output quality, when measured with respect to output pathologies such as counterfactual and logically-flawed statements, and general failures like not staying on topic.
• Overall, 80.0% of the outputs evaluated contained memorized data, but outputs containing the most memorized content were also more likely to be considered of high quality.