COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domain

Understand

Large Language Models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence (AI) technology which is rapidly evolving and promises to aid in medical diagnosis either by assisting doctors or by simulating a doctor's workflow in more advanced and complex implementations.

• In this technical paper, we outline Cognitive Network Evaluation Toolkit for Medical Domains (COGNET-MD), which constitutes a novel benchmark for LLM evaluation in the medical domain.
• Specifically, we propose a scoring-framework with increased difficulty to assess the ability of LLMs in interpreting medical text.
• The proposed framework is accompanied with a database of Multiple Choice Quizzes (MCQs).