Probing the State of the Art: A Critical Look at Visual Representation Evaluation

Understand

Self-supervised research improved greatly over the past half decade, with much of the growth being driven by objectives that are hard to quantitatively compare.

• These techniques include colorization, cyclical consistency, and noise-contrastive estimation from image patches.
• Consequently, the field has settled on a handful of measurements that depend on linear probes to adjudicate which approaches are the best.
• Our first contribution is to show that this test is insufficient and that models which perform poorly (strongly) on linear classification can perform strongly (weakly) on more involved tasks like temporal activity localization.