Discovering Latent Knowledge in Language Models Without Supervision

Understand

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect.

• We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.
• Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations.
• It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.