Fetching the paper…
Reading the bibliography…
Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data.
Does Learning Require Memorization? A Short Tale about a Long Tail, January 2021
Vitaly Feldman · 1906
Earlier work this paper cites.
The interpretation of interaction in contingency tables
Edward H Simpson · 1951
Earlier work this paper cites.
A method for the construction of minimum-redundancy codes
David A Huffman · 1952
Earlier work this paper cites.
Binary codes capable of correcting deletions, insertions, and reversals
Vladimir I Levenshtein et al · 1966
Earlier work this paper cites.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler · 2015
Earlier work this paper cites.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Xiaodong Song · 2018
Earlier work this paper cites.
Benign overfitting in linear regression
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler · 2020
Earlier work this paper cites.
Extracting training data from large language models
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel · 2020
Earlier work this paper cites.
The pile: An 800gb dataset of diverse text for language modeling
Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy · 2020
Earlier work this paper cites.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf · 2020
Earlier work this paper cites.
Deduplicating training data makes language models better
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini · 2021
Earlier work this paper cites.
Counterfactual memorization in neural language models
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini · 2021
Cited alongside, same era.
What does it mean for a language model to preserve privacy?
Hannah Brown, Katherine Lee, FatemehSadat Mireshghallah, R. Shokri, and Florian Tramèr · 2022
Cited alongside, same era.
Preventing verbatim memorization in language models gives a false sense of privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini · 2022
Cited alongside, same era.
Deduplicating training data mitigates privacy risks in language models
Nikhil Kandpal, Eric Wallace, and Colin Raffel · 2022
Cited alongside, same era.
Benign, tempered, or catastrophic: A taxonomy of overfitting, 2022
Measures of information reflect memorization patterns, 2023
Rachit Bansal, Danish Pruthi, and Yonatan Belinkov · 2023
Later among the works it cites.
Quantifying Memorization Across Neural Language Models, March 2023
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang · 2023
Later among the works it cites.
Verna Dankers, Ivan Titov, and Dieuwke Hupkes · 2023
Later among the works it cites.
SoK: Memorization in general-purpose large language models, 2023
Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Preetum Nakkiran · 2022
Cited alongside, same era.
An empirical analysis of memorization in fine-tuned autoregressive language models
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick · 2022
Cited alongside, same era.
Beyond neural scaling laws: beating power law scaling via data pruning
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos · 2022
Cited alongside, same era.
Defending against reconstruction attacks with rényi differential privacy, 2022
Pierre Stock, Igor Shilov, Ilya Mironov, and Alexandre Sablayrolles · 2022
Cited alongside, same era.
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan · 2022
Cited alongside, same era.
Provably confidential language modelling
Xuandong Zhao, Lei Li, and Yu-Xiang Wang · 2022
Cited alongside, same era.
Emergent and predictable memorization in large language models
Stella Rose Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G. Anthony, Shivanshu Purohit, and Edward Raf
Cited in the paper.
Pythia: A suite for analyzing large language models across training and scaling
Stella Rose Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal
Cited in the paper.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah · 2023
Later among the works it cites.
Copyright violations and large language models
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Sogaard · 2023
Later among the works it cites.
Detecting pretraining data from large language models
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer · 2023
Later among the works it cites.
Copyright traps for large language models
Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye · 2024
Closest in time.
Rethinking llm memorization through the lens of adversarial compression, 2024
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter · 2024
Closest in time.
Unveiling memorization in code models
Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo · 2024
Closest in time.