Surgical Text-to-Image Generation

Understand

Acquiring surgical data for research and development is significantly hindered by high annotation costs and practical and ethical constraints.

• Utilizing synthetically generated images could offer a valuable alternative.
• In this work, we explore adapting text-to-image generative models for the surgical domain using the CholecT50 dataset, which provides surgical images annotated with action triplets (instrument, verb, target).
• We investigate several language models and find T5 to offer more distinct features for differentiating surgical actions on triplet-based textual inputs, and showcasing stronger alignment between long and triplet-based captions.