Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences

Understand

In this paper, we consider a novel task, Spatio-Temporal Video Grounding for Multi-Form Sentences (STVG).

• Given an untrimmed video and a declarative/interrogative sentence depicting an object, STVG aims to localize the spatio-temporal tube of the queried object.
• STVG has two challenging settings: (1) We need to localize spatio-temporal object tubes from untrimmed videos, where the object may only exist in a very small segment of the video; (2) We deal with multi-form sentences, including the declarative sentences with explicit objects and interrogative sentences with unknown objects.
• Existing methods cannot tackle the STVG task due to the ineffective tube pre-generation and the lack of object relationship modeling.