Estimating Knowledge in Large Language Models Without Generating a Single Token

The importance of nonverbal communication in the courtroom

Martin S Remland. 1994 · 1994

Earlier work this paper cites.

Okapi at trec-3

Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995 · 1995

Earlier work this paper cites.

Evaluating content selection in summarization: The pyramid method

Ani Nenkova and Rebecca Passonneau. 2004 · 2004

Earlier work this paper cites.

A survey of named entity recognition and classification

David Nadeau and Satoshi Sekine. 2007 · 2007

Earlier work this paper cites.

Wikidata: a free collaborative knowledgebase

Denny Vrandečić and Markus Krötzsch. 2014 · 2014

Earlier work this paper cites.

Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017 · 2017

Earlier work this paper cites.

Understanding intermediate layers using linear classifier probes

Guillaume Alain and Yoshua Bengio. 2017 · 2017

Earlier work this paper cites.

What do neural machine translation models learn about morphology?

Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

User’s guide to correlation coefficients

Haldun Akoglu. 2018 · 2018

Earlier work this paper cites.

Correlation coefficients: Appropriate use and interpretation

Patrick Schober, Christa Boer, and Lothar Schwarte. 2018 · 2018

Earlier work this paper cites.

A structural probe for finding syntax in word representations

John Hewitt and Christopher D. Manning. 2019 · 2019

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019 · 2019

Earlier work this paper cites.

Language models as knowledge bases?

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Crowdsourcing lightweight pyramids for manual summary evaluation

Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019 · 2019

Earlier work this paper cites.

On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers

Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020 · 2020

Earlier work this paper cites.

interpreting gpt: the logit lens

nostalgebraist. 2020 · 2020

Earlier work this paper cites.

How much knowledge can you pack into the parameters of a language model?

Adam Roberts, Colin Raffel, and Noam Shazeer. 2020 · 2020

Earlier work this paper cites.

Evaluating entity disambiguation and the role of popularity in retrieval-based NLP

Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021 · 2021

Earlier work this paper cites.

Transformer feed-forward layers are key-value memories

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021 · 2021

Earlier work this paper cites.

Implicit representations of meaning in neural language models

Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021 · 2021

Cited alongside, same era.

Finding a balanced degree of automation for summary evaluation

Shiyue Zhang and Mohit Bansal. 2021 · 2021

Cited alongside, same era.

Attributed question answering: Evaluation and modeling for attributed large language models

Original

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022 · 2022

Cited alongside, same era.

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Cited alongside, same era.

A token-level reference-free hallucination detection benchmark for free-form text generation

Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022 · 2022

Cited alongside, same era.

FActScore: Fine-grained atomic evaluation of factual precision in long form text generation

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023 · 2023

Later among the works it cites.

Investigating semantic subspaces of transformer sentence embeddings through linear structural probing

Dmitry Nikolaev and Sebastian Padó. 2023 · 2023

Later among the works it cites.

On early detection of hallucinations in factual question answering

Original

Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. 2023 · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023 · 2023