Unified Multimodal Interleaved Document Representation for Retrieval

Understand

Information Retrieval (IR) methods aim to identify documents relevant to a query, which have been widely applied in various natural language tasks.

• However, existing approaches typically consider only the textual content within documents, overlooking the fact that documents can contain multiple modalities, including images and tables.
• Also, they often segment each long document into multiple discrete passages for embedding, which prevents them from capturing the overall document context and interactions between paragraphs.
• To address these two challenges, we propose a method that holistically embeds documents interleaved with multiple modalities by leveraging the capability of recent vision-language models that enable the processing and integration of text, images, and tables into a unified format and representation.