The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem

Understand

Large language models (LLMs) are useful tools with the capacity for performing specific types of knowledge work at an effective scale.

• However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of ``hallucination,'' where LLMs can generate fabricated information.
• This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm.
• To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, and potentially applicable to other medical safety-critical contexts.