Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

Understand

Purpose: To develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports using open-weights large language models (LMs) and retrieval augmented generation (RAG), and to assess the effects of model configuration variables on extraction performance.

• Methods and Materials: The study utilized two datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status.
• An automated pipeline was developed to benchmark the performance of various LMs and RAG configurations.
• The impact of model size, quantization, prompting strategies, output formatting, and inference parameters was systematically evaluated.