Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification

Understand

Recent advances in large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications.

• This study investigates the performance of LLMs such as the ChatGPT family of models (GPT-3.5s, GPT-4) in biomedical tasks beyond question-answering.
• Because no patient data can be passed to the OpenAI API public interface, we evaluated model performance with over 10000 samples as proxies for two fundamental tasks in the clinical domain - classification and reasoning.
• The first task is classifying whether statements of clinical and policy recommendations in scientific literature constitute health advice.