Multimodal Markup Document Models for Graphic Design Completion

Understand

We introduce MarkupDM, a multimodal markup document model that represents graphic design as an interleaved multimodal document consisting of both markup language and images.

• Unlike existing holistic approaches that rely on an element-by-attribute grid representation, our representation accommodates variable-length elements, type-dependent attributes, and text content.
• Inspired by fill-in-the-middle training in code generation, we train the model to complete the missing part of a design document from its surrounding context, allowing it to treat various design tasks in a unified manner.
• Our model also supports image generation by predicting discrete image tokens through a specialized tokenizer with support for image transparency.