What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

How context affects language models’ factual predictions

Original

Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020 · 2005

Earlier work this paper cites.

T-REx: A large scale alignment of natural language with knowledge base triples

Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Language models as knowledge bases?

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019 · 2019

Earlier work this paper cites.

Calibration of pre-trained transformers

Shrey Desai and Greg Durrett. 2020 · 2020

Earlier work this paper cites.

How can we know what language models know?

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020 · 2020

Earlier work this paper cites.

Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly

Nora Kassner and Hinrich Schütze. 2020 · 2020

Earlier work this paper cites.

Syntactic similarity of the sentences in a multi-lingual parallel corpus based on the Euclidean distance of their dependency trees

Masanori Oya. 2020 · 2020

Cited alongside, same era.

Measuring and improving consistency in pretrained language models

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021 · 2021

Cited alongside, same era.

Sorting through the noise: Testing robustness of information processing in pre-trained language models

Lalchand Pandia and Allyson Ettinger. 2021 · 2021

Cited alongside, same era.

Calibrate before use: Improving few-shot performance of language models

Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 · 2021

Cited alongside, same era.

Factual consistency of multilingual pretrained language models

Constanza Fierro and Anders Søgaard. 2022 · 2022

Cited alongside, same era.

DLAMA: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models

Amr Keleg and Walid Magdy. 2023 · 2023

Later among the works it cites.

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 · 2023

Later among the works it cites.

Siren’s song in the ai ocean: A survey on hallucination in large language models

Original

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023 · 2023

Later among the works it cites.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, et al. 2024 · 2024

Closest in time.