Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit

Understand

Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse.

• However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored.
• In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits.
• We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location.