2022

Large Language Models Struggle to Learn Long-Tail Knowledge

Kandpal, Nikhil, Deng, Haikang, Roberts, Adam et al.

Understand

The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models.

However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely.
In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web.
In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training.

Built on

The probabilistic relevance framework: BM25 and beyond
Robertson, S. and Zaragoza, H · 2009
Earlier work this paper cites.
DBpedia Spotlight: Shedding light on the web of documents
Mendes, P. N., Jakob, M., García-Silva, A., and Bizer, C · 2011
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P · 2016
Earlier work this paper cites.
TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
T-REx: A large scale alignment of natural language with knowledge base triples
Elsahar, H., Vougiouklis, P., Remaci, A., Gravier, C., Hare, J., Laforest, F., and Simperl, E · 2018
Earlier work this paper cites.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D · 2019
Earlier work this paper cites.
Openwebtext corpus, 2019
Gokaslan, A. and Cohen, V · 2019
Earlier work this paper cites.
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Rhinehart, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., et al · 2019
Earlier work this paper cites.
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., and Toutanova, K · 2019
Earlier work this paper cites.
Language models as knowledge bases?
Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S · 2019
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
Entities as experts: Sparse memory access with entity supervision
Févry, T., Soares, L. B., FitzGerald, N., Choi, E., and Kwiatkowski, T · 2020
Earlier work this paper cites.
The Pile: An 800GB dataset of diverse text for language modeling
Original
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al · 2020
Earlier work this paper cites.
Retrieval augmented language model pre-training
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M · 2020
Earlier work this paper cites.

Similar

Dense passage retrieval for open-domain question answering
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t · 2020
Cited alongside, same era.
Are pretrained language models symbolic reasoners over knowledge?
Kassner, N., Krojer, B., and Schütze, H · 2020
Cited alongside, same era.
Retrieval-augmented generation for knowledge-intensive NLP tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al · 2020
Cited alongside, same era.
How context affects language models’ factual predictions
Petroni, F., Lewis, P. S. H., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, S · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al · 2020
Cited alongside, same era.
How much knowledge can you pack into the parameters of a language model?
Roberts, A., Raffel, C., and Shazeer, N · 2020
Cited alongside, same era.
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021
Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S · 2021
Cited alongside, same era.
Extracting training data from large language models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C · 2021
Cited alongside, same era.
Autoregressive entity retrieval
De Cao, N., Izacard, G., Riedel, S., and Petroni, F · 2021
Cited alongside, same era.
Distilling knowledge from reader to retriever for question answering
Izacard, G. and Grave, E · 2021
Cited alongside, same era.
Deduplicating training data makes language models better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N · 2021
Cited alongside, same era.
Msa transformer
Rao, R. M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A · 2021
Cited alongside, same era.

Then

Tracing knowledge in language models back to the training data
Akyürek, E., Bolukbasi, T., Liu, F., Xiong, B., Tenney, I., Andreas, J., and Guu, K · 2022
Closest in time.
GPT-Neox-20B: An open-source autoregressive language model
Original
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al · 2022
Closest in time.
Measuring causal effects of data statistics on language model’s factual predictions
Original
Elazar, Y., Kassner, N., Ravfogel, S., Feder, A., Ravichander, A., Mosbach, M., Belinkov, Y., Schütze, H., and Goldberg, Y · 2022
Closest in time.
ORCA: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data
Original
Han, X. and Tsvetkov, Y · 2022
Closest in time.
Preventing verbatim memorization in language models gives a false sense of privacy
Original
Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., and Carlini, N · 2022
Closest in time.
Deduplicating training data mitigates privacy risks in language models
Kandpal, N., Wallace, E., and Raffel, C · 2022
Closest in time.
The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., et al · 2022
Closest in time.
Challenges in generalization in open domain question answering
Liu, L., Lewis, P., Riedel, S., and Stenetorp, P · 2022
Closest in time.
When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories
Original
Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D · 2022
Closest in time.
Impact of pretraining term frequencies on few-shot reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S · 2022
Closest in time.
BLOOM: A 176b-parameter open-access multilingual language model
Original
Scao, T. L., Fan, A., Akiki, C., Pavlick, E.-J., Ili’c, S., Hesslow, D., Castagn’e, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., et al · 2022
Closest in time.
On the effect of pretraining corpora on in-context learning by a large-scale language model
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al · 2022
Closest in time.
What language model architecture and pretraining objective work best for zero-shot generalization?
Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C · 2022
Closest in time.
Measuring forgetting of memorized training examples
Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…