2023

Challenges with unsupervised LLM knowledge discovery

Farquhar, Sebastian, Varma, Vikrant, Kenton, Zachary et al.

Understand

We show that existing unsupervised methods on large language model (LLM) activations do not discover knowledge -- instead they seem to discover whatever feature of the activations is most prominent.

The idea behind unsupervised knowledge elicitation is that knowledge satisfies a consistency structure, which can be used to discover knowledge.
We first prove theoretically that arbitrary features (not just knowledge) satisfy the consistency structure of a particular leading unsupervised knowledge-elicitation method, contrast-consistent search (Burns et al.
- arXiv:2212.03827).

Built on

DBpedia: A nucleus for a web of open data
S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives · 2007
Earlier work this paper cites.
Learning word vectors for sentiment analysis
A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts · 2011
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Understanding intermediate layers using linear classifier probes
G. Alain and Y. Bengio · 2016
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2018
Earlier work this paper cites.
BoolQ: Exploring the surprising difficulty of natural Yes/No questions
C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova · 2019
Earlier work this paper cites.
Challenging common assumptions in the unsupervised learning of disentangled representations
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem · 2019
Earlier work this paper cites.
Haiku: Sonnet for JAX, 2020
T. Hennigan, T. Cai, T. Norman, L. Martens, and I. Babuschkin · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2020
Earlier work this paper cites.
A general language assistant as a laboratory for alignment
A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan · 2021
Earlier work this paper cites.
On the dangers of stochastic parrots: Can language models be too big?
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell · 2021
Earlier work this paper cites.
Eliciting latent knowledge: How to tell if your eyes deceive you, Dec. 2021
P. Christiano, A. Cotra, and M. Xu · 2021
Earlier work this paper cites.
Truthful AI: Developing and governing AI that does not lie
Original
O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders · 2021
Earlier work this paper cites.
Alignment of language agents
Original
Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving · 2021
Earlier work this paper cites.

Similar

TruthfulQA: Measuring how models mimic human falsehoods
Original
S. Lin, J. Hilton, and O. Evans · 2021
Cited alongside, same era.
How “discovering latent knowledge in language models without supervision” fits into a broader alignment scheme
C. Burns · 2022
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Original
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al · 2022
Cited alongside, same era.
Scaling instruction-finetuned language models
Original
H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al · 2022
Cited alongside, same era.
Discovering latent concepts learned in bert
Original
F. Dalvi, A. R. Khan, F. Alam, N. Durrani, J. Xu, and H. Sajjad · 2022
Cited alongside, same era.
Toy models of superposition
N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah · 2022
Cited alongside, same era.
Training compute-optimal large language models
Original
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al · 2022
Cited alongside, same era.
My reservations about discovering latent knowledge
R. Huben · 2022
Cited alongside, same era.
Talking about large language models
M. Shanahan · 2022
Cited alongside, same era.
The internal state of an LLM knows when its lying
A. Azaria and T. Mitchell · 2023
Cited alongside, same era.
Eliciting latent predictions from transformers with the tuned lens
Original
N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt · 2023
Cited alongside, same era.
Sparks of artificial general intelligence: Early experiments with GPT-4
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang · 2023
Cited alongside, same era.

Then

Contrast pairs drive the empirical performance of contrast consistent search (ccs), May 2023
S. Emmons · 2023
Closest in time.
Comparing optimization targets for contrast-consistent search
Original
H. Fry, S. Fallows, I. Fan, J. Wright, and N. Schoots · 2023
Closest in time.
Methods for measuring, updating, and visualizing factual beliefs in language models
P. Hase, M. Diab, A. Celikyilmaz, X. Li, Z. Kozareva, V. Stoyanov, M. Bansal, and S. Iyer · 2023
Closest in time.
Still no lie detector for language models: Probing empirical and conceptual roadblocks
Original
B. Levinstein and D. A. Herrmann · 2023
Closest in time.
Inference-Time intervention: Eliciting truthful answers from a language model
K. Li, O. Patel, F. Viegas, H. Pfister, and M. Wattenberg · 2023
Closest in time.
The geometry of truth: Emergent linear structure in large language model representations of True/False datasets
S. Marks and M. Tegmark · 2023
Closest in time.
Gpt-4 technical report
R. OpenAI · 2023
Closest in time.
How to catch an AI liar: Lie detection in Black-Box LLMs by asking unrelated questions
L. Pacchiardi, A. J. Chan, S. Mindermann, I. Moscovitz, A. Y. Pan, Y. Gal, O. Evans, and J. Brauner · 2023
Closest in time.
AI deception: A survey of examples, risks, and potential solutions
P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks · 2023
Closest in time.
What discovering latent knowledge did and did not find, Mar. 2023
F. Roger · 2023
Closest in time.
Strategically deceive their users when put under pressure
J. Scheurer, M. Balesni, and M. Hobbhahn · 2023
Closest in time.
Role-play with large language models
Original
M. Shanahan, K. McDonell, and L. Reynolds · 2023
Closest in time.
Gaussian process probes (gpp) for uncertainty-aware probing
Original
Z. Wang, A. Ku, J. Baldridge, T. L. Griffiths, and B. Kim · 2023
Closest in time.
Representation engineering: A Top-Down approach to AI transparency
A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Zico Kolter, and D. Hendrycks · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…