Grounding Language Models for Visual Entity Recognition

Understand

We introduce AutoVER, an Autoregressive model for Visual Entity Recognition.

• Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation.
• It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning.
• Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever.