FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks

Understand

Advancements in Multimodal Large Language Models (MLLMs) have significantly improved medical task performance, such as Visual Question Answering (VQA) and Report Generation (RG).

• However, the fairness of these models across diverse demographic groups remains underexplored, despite its importance in healthcare.
• This oversight is partly due to the lack of demographic diversity in existing medical multimodal datasets, which complicates the evaluation of fairness.
• In response, we propose FMBench, the first benchmark designed to evaluate the fairness of MLLMs performance across diverse demographic attributes.