ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks

Understand

While some work attempt to generate front-end code intelligently from UI screenshots, it may be more convenient to utilize UI design drafts in Sketch which is a popular UI design software, because we can access multimodal UI information directly such as layers type, position, size, and visual images.

• However, fragmented layers could degrade the code quality without being merged into a whole part if all of them are involved in the code generation.
• In this paper, we propose a pipeline to merge fragmented layers automatically.
• We first construct a graph representation for the layer tree of a UI draft and detect all fragmented layers based on the visual features and graph neural networks.