Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery

Understand

Despite growing interest in using large language models (LLMs) in healthcare, current explorations do not assess the real-world utility and safety of LLMs in clinical settings.

• Our objective was to determine whether two LLMs can serve information needs submitted by physicians as questions to an informatics consultation service in a safe and concordant manner.
• Sixty six questions from an informatics consult service were submitted to GPT-3.5 and GPT-4 via simple prompts.
• 12 physicians assessed the LLM responses' possibility of patient harm and concordance with existing reports from an informatics consultation service.