DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ

Understand

Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy.

• Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex.
• To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures.
• To achieve this, we create three new datasets: DaTikZv2, the largest TikZ dataset to date, containing over 360k human-created TikZ graphics; SketchFig, a dataset that pairs hand-drawn sketches with their corresponding scientific figures; and MetaFig, a collection of diverse scientific figures and associated metadata.