LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

Understand

Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images.

• Meanwhile, current general-domain multimodal models for videos still lack the capabilities to understand and engage in conversations about surgical videos.
• One major contributing factor is the absence of datasets in the surgical field.
• In this paper, we create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs, the largest of its kind so far.