TubeDETR: Spatio-Temporal Video Grounding with Transformers

Understand

We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query.

• This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions.
• To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection.
• Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization.