Video Instance Segmentation via Multi-scale Spatio-temporal Split Attention Transformer

Understand

State-of-the-art transformer-based video instance segmentation (VIS) approaches typically utilize either single-scale spatio-temporal features or per-frame multi-scale features during the attention computations.

• We argue that such an attention computation ignores the multi-scale spatio-temporal feature relationships that are crucial to tackle target appearance deformations in videos.
• To address this issue, we propose a transformer-based VIS framework, named MS-STS VIS, that comprises a novel multi-scale spatio-temporal split (MS-STS) attention module in the encoder.
• The proposed MS-STS module effectively captures spatio-temporal feature relationships at multiple scales across frames in a video.