VCoder: Versatile Vision Encoders for Multimodal Large Language Models

Understand

Humans possess the remarkable skill of Visual Perception, the ability to see and understand the seen, helping them make sense of the visual world and, in turn, reason.

• Multimodal Large Language Models (MLLM) have recently achieved impressive performance on vision-language tasks ranging from visual question-answering and image captioning to visual reasoning and image generation.
• However, when prompted to identify or count (perceive) the entities in a given image, existing MLLM systems fail.
• Working towards developing an accurate MLLM system for perception and reasoning, we propose using Versatile vision enCoders (VCoder) as perception eyes for Multimodal LLMs.