X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks

Understand

In this paper, we study the challenging instance-wise vision-language tasks, where the free-form language is required to align with the objects instead of the whole image.

• To address these tasks, we propose X-DETR, whose architecture has three major components: an object detector, a language encoder, and vision-language alignment.
• The vision and language streams are independent until the end and they are aligned using an efficient dot-product operation.
• The whole network is trained end-to-end, such that the detector is optimized for the vision-language tasks instead of an off-the-shelf component.