Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

Understand

Large language and multimodal models have shown remarkable success on various benchmarks focused on specific skills such as general-purpose programming, math word problem-solving, and visual question answering.

• However, it is unclear how well these models perform on tasks that require a combination of these skills.
• In this paper, we curate a novel program synthesis benchmark based on the real-world tasks in the XLogoOnline visual programming environment.
• Each task requires a combination of different skills such as spatial planning, basic programming, and logical reasoning.