CLIP-GCD: Simple Language Guided Generalized Category Discovery

Understand

Generalized Category Discovery (GCD) requires a model to both classify known categories and cluster unknown categories in unlabeled data.

• Prior methods leveraged self-supervised pre-training combined with supervised fine-tuning on the labeled data, followed by simple clustering methods.
• In this paper, we posit that such methods are still prone to poor performance on out-of-distribution categories, and do not leverage a key ingredient: Semantic relationships between object categories.
• We therefore propose to leverage multi-modal (vision and language) models, in two complementary ways.