ClusterFit: Improving Generalization of Visual Representations

Understand

Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks.

• However, due to the lack of strong discriminative signals, these learned representations may overfit to the pre-training objective (e.g., hashtag prediction) and not generalize well to downstream tasks.
• In this work, we present a simple strategy - ClusterFit (CF) to improve the robustness of the visual representations learned during pre-training.
• Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels.