Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale

Understand

Large language models (LLMs) are increasingly used for analytical tasks, yet their effectiveness in real-world applications remains underexamined, partly due to the opacity of proprietary models.

• We evaluate ChatGPT (GPT-3.5 and GPT-4) on the practical task of extracting research challenges from a large scholarly corpus in Human-Computer Interaction (HCI).
• Using a two-step approach, we first apply GPT-3.5 to extract candidate challenges from the 879 papers in the 2023 ACM CHI Conference proceedings, then use GPT-4 to select the most relevant challenges per paper.
• This process yielded 4,392 research challenges across 113 topics, which we organized through topic modeling and present in an interactive visualization.