Fetching the paper…
Reading the bibliography…
Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data.
Random number generation and quasi-Monte Carlo methods
H. Niederreiter · 1992
Earlier work this paper cites.
Pointer sentinel mixture models, 2016
S. Merity, C. Xiong, J. Bradbury, and R. Socher · 2016
Earlier work this paper cites.
The secret sharer: Evaluating and testing unintended memorization in neural networks
N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song · 2019
Earlier work this paper cites.
Generalization through memorization: Nearest neighbor language models
U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis · 2019
Earlier work this paper cites.
Language models as knowledge bases?
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al · 2019
Earlier work this paper cites.
Auditing data provenance in text-generation models
C. Song and V. Shmatikov · 2019
Earlier work this paper cites.
Language models are few-shot learners
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al · 2020
Earlier work this paper cites.
Retrieval augmented language model pre-training
K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang · 2020
Earlier work this paper cites.
Extracting training data from large language models
N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al · 2021
Cited alongside, same era.
How bpe affects memorization in transformers
E. Kharitonov, M. Baroni, and D. Hupkes · 2021
Cited alongside, same era.
Counterfactual memorization in neural language models
C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini · 2021
Cited alongside, same era.
A review on language models as knowledge bases
B. AlKhamissi, M. Li, A. Celikyilmaz, M. Diab, and M. Ghazvininejad · 2022
Cited alongside, same era.
Improving language models by retrieving from trillions of tokens
S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al · 2022
Transformer memory as a differentiable search index
Y. Tay, V. Tran, M. Dehghani, J. Ni, D. Bahri, H. Mehta, Z. Qin, K. Hui, Z. Zhao, J. Gupta, et al · 2022
Later among the works it cites.
Memorization without overfitting: Analyzing the training dynamics of large language models
K. Tirumala, A. Markosyan, L. Zettlemoyer, and A. Aghajanyan · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al · 2022
Later among the works it cites.
Large language models with controllable working memory
D. Li, A. S. Rawat, M. Zaheer, X. Wang, M. Lukasik, A. Veit, F. Yu, and S. Kumar · 2023
Later among the works it cites.
Analyzing leakage of personally identifiable information in language models
N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Quantifying memorization across neural language models
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang · 2022
Cited alongside, same era.
Understanding transformer memorization recall through idioms
A. Haviv, I. Cohen, J. Gidron, R. Schuster, Y. Goldberg, and M. Geva · 2022
Cited alongside, same era.
Measuring forgetting of memorized training examples
M. Jagielski, O. Thakkar, F. Tramer, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot, et al · 2022
Cited alongside, same era.
The curious case of absolute position embeddings
K. Sinha, A. Kazemnejad, S. Reddy, J. Pineau, D. Hupkes, and A. Williams · 2022
Cited alongside, same era.
Emergent and predictable memorization in large language models
S. Biderman, U. S. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raf
Cited in the paper.
Pythia: A suite for analyzing large language models across training and scaling
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al
Cited in the paper.
Textbooks are all you need ii: phi-1.5
Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee
Cited in the paper.
Membership inference attacks against language models via neighbourhood comparison
J. Mattern, F. Mireshghallah, Z. Jin, B. Schölkopf, M. Sachan, and T. Berg-Kirkpatrick · 2023
Later among the works it cites.
How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven
R. T. McCoy, P. Smolensky, T. Linzen, J. Gao, and A. Celikyilmaz · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al · 2023
Later among the works it cites.
Roformer: Enhanced transformer with rotary position embedding
J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu · 2024
Closest in time.