Fetching the paper…
Reading the bibliography…
While large language models (LLMs) are extensively used, there are raising concerns regarding privacy, security, and copyright due to their opaque training data, which brings the problem of detecting pre-training data on the table.
A mathematical theory of communication
C. E. Shannon · 1948
Earlier work this paper cites.
Gutenbergpy
R. Angelescu · 2013
Earlier work this paper cites.
Membership inference attacks against machine learning models
R. Shokri, M. Stronati, C. Song, and V. Shmatikov · 2017
Earlier work this paper cites.
Privacy risk in machine learning: Analyzing the connection to overfitting
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2019
Earlier work this paper cites.
The pile: An 800gb dataset of diverse text for language modeling
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al · 2020
Earlier work this paper cites.
Privacy risks of general-purpose language models
X. Pan, M. Zhang, S. Ji, and M. Yang · 2020
Earlier work this paper cites.
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021
S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman · 2021
Earlier work this paper cites.
Label-only membership inference attacks
C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot · 2021
Earlier work this paper cites.
Documenting large webtext corpora: A case study on the colossal clean crawled corpus
J. Dodge, A. Marasovic, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner · 2021
Earlier work this paper cites.
Differentially private fine-tuning of language models
D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, et al · 2021
Earlier work this paper cites.
Language contamination helps explains the cross-lingual capabilities of english pretrained models
T. Blevins and L. Zettlemoyer · 2022
Earlier work this paper cites.
What does it mean for a language model to preserve privacy?
H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr · 2022
Earlier work this paper cites.
Membership inference attacks from first principles
N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer · 2022
Earlier work this paper cites.
Membership inference attacks on machine learning: A survey
H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang · 2022
Earlier work this paper cites.
Data contamination: From memorization to exploitation
I. Magar and R. Schwartz · 2022
Earlier work this paper cites.
Quantifying privacy risks of masked language models using membership inference attacks
F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri · 2022
Cited alongside, same era.
Lamda: Language models for dialog applications
R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, et al · 2022
Cited alongside, same era.
Memorization without overfitting: Analyzing the training dynamics of large language models
K. Tirumala, A. Markosyan, L. Zettlemoyer, and A. Aghajanyan · 2022
Cited alongside, same era.
On the importance of difficulty calibration in membership inference attacks
L. Watson, C. Guo, G. Cormode, and A. Sablayrolles · 2022
Cited alongside, same era.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al · 2023
Membership inference attacks against language models via neighbourhood comparison
J. Mattern, F. Mireshghallah, Z. Jin, B. Schoelkopf, M. Sachan, and T. Berg-Kirkpatrick · 2023
Later among the works it cites.
Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities
M. Mozes, X. He, B. Kleinberg, and L. D. Griffin · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models, 2023
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample · 2023
Later among the works it cites.
On provable copyright protection for generative models
N. Vyas, S. M. Kakade, and B. Barak · 2023
Later among the works it cites.
CodeIPPrompt: Intellectual property infringement assessment of code language models
Z. Yu, Y. Wu, N. Zhang, C. Wang, Y. Vorobeychik, and C. Xiao · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Pythia: A suite for analyzing large language models across training and scaling
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al · 2023
Cited alongside, same era.
Quantifying memorization across neural language models
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang · 2023
Cited alongside, same era.
Speak, memory: An archaeology of books known to chatGPT/GPT-4
K. K. Chang, M. H. Cramer, S. Soni, and D. Bamman · 2023
Cited alongside, same era.
Towards next-generation intelligent assistants leveraging llm techniques
X. L. Dong, S. Moon, Y. E. Xu, K. Malik, and Z. Yu · 2023
Cited alongside, same era.
Sok: Memorization in general-purpose large language models
V. Hartmann, A. Suri, V. Bindschaedler, D. Evans, S. Tople, and R. West · 2023
Cited alongside, same era.
Foundation models and fair use, 2023
P. Henderson, X. Li, D. Jurafsky, T. Hashimoto, M. A. Lemley, and P. Liang · 2023
Cited alongside, same era.
Preventing verbatim memorization in language models gives a false sense of privacy, 2023
D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini · 2023
Cited alongside, same era.
How to protect copyright data in optimization of large language models?
T. Chu, Z. Song, and C. Yang · 2024
Closest in time.
Do membership inference attacks work on large language models?
M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi · 2024
Closest in time.
Time travel in LLMs: Tracing data contamination in large language models
S. Golchin and M. Surdeanu · 2024
Closest in time.
Olmo: Accelerating the science of language models
D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. D. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. A. Smith, and H. Hajishirzi · 2024
Closest in time.
Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator
H. Huang, Y. Feng, C. Shi, L. Xu, J. Yu, and S. Yang · 2024
Closest in time.
Propile: Probing privacy leakage in large language models
S. Kim, S. Yun, H. Lee, M. Gubri, S. Yoon, and S. J. Oh · 2024
Closest in time.
Proving test set contamination in black-box language models
Y. Oren, N. Meister, N. S. Chatterji, F. Ladhak, and T. Hashimoto · 2024
Closest in time.
Visual adversarial examples jailbreak aligned large language models
X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal · 2024
Closest in time.
Detecting pretraining data from large language models
W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer · 2024
Closest in time.
A survey on large language model (llm) security and privacy: The good, the bad, and the ugly
Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang · 2024
Closest in time.