Can Visual Foundation Models Achieve Long-term Point Tracking?

Understand

Large-scale vision foundation models have demonstrated remarkable success across various tasks, underscoring their robust generalization capabilities.

• While their proficiency in two-view correspondence has been explored, their effectiveness in long-term correspondence within complex environments remains unexplored.
• To address this, we evaluate the geometric awareness of visual foundation models in the context of point tracking: (i) in zero-shot settings, without any training; (ii) by probing with low-capacity layers; (iii) by fine-tuning with Low Rank Adaptation (LoRA).
• Our findings indicate that features from Stable Diffusion and DINOv2 exhibit superior geometric correspondence abilities in zero-shot settings.