Cluster-norm for Unsupervised Probing of Knowledge

Understand

The deployment of language models brings challenges in generating reliable information, especially when these models are fine-tuned using human preferences.

• To extract encoded knowledge without (potentially) biased human labels, unsupervised probing techniques like Contrast-Consistent Search (CCS) have been developed (Burns et al., 2022).
• However, salient but unrelated features in a given dataset can mislead these probes (Farquhar et al., 2023).
• Addressing this, we propose a cluster normalization method to minimize the impact of such features by clustering and normalizing activations of contrast pairs before applying unsupervised probing techniques.