Knowing Where and What: Unified Word Block Pretraining for Document Understanding

Understand

Due to the complex layouts of documents, it is challenging to extract information for documents.

• Most previous studies develop multimodal pre-trained models in a self-supervised way.
• In this paper, we focus on the embedding learning of word blocks containing text and layout information, and propose UTel, a language model with Unified TExt and Layout pre-training.
• Specifically, we propose two pre-training tasks: Surrounding Word Prediction (SWP) for the layout learning, and Contrastive learning of Word Embeddings (CWE) for identifying different word blocks.