Uncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentation

Understand

Self-supervised pre-training strategies have recently shown impressive results for training general-purpose feature extraction backbones in computer vision.

• In combination with the Vision Transformer architecture, the DINO self-distillation technique has interesting emerging properties, such as unsupervised clustering in the latent space and semantic correspondences of the produced features without using explicit human-annotated labels.
• The STEGO method for unsupervised semantic segmentation contrastively distills feature correspondences of a DINO-pre-trained Vision Transformer and recently set a new state of the art.
• However, the detailed workings of STEGO have yet to be disentangled, preventing its usage in safety-critical applications.