Fetching the paper…
Reading the bibliography…
Whether LLMs memorize their training data and what this means, from measuring privacy leakage to detecting copyright violations, has become a rapidly growing area of research.
On bootstrapping the roc curve
Patrice Bertail, Stéphan Clémençcon, and Nicolas Vayatis · 2008
Earlier work this paper cites.
Regression discontinuity designs in economics
David S Lee and Thomas Lemieux · 2010
Earlier work this paper cites.
Character-level convolutional networks for text classification
Xiang Zhang, Junbo Zhao, and Yann LeCun · 2015
Earlier work this paper cites.
Logan: Membership inference attacks against generative models
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin · 2018
Earlier work this paper cites.
Ethical challenges in data-driven dialogue systems
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau · 2018
Earlier work this paper cites.
Understanding membership inferences on well-generalized learning models
Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen · 2018
Earlier work this paper cites.
Machine learning with membership privacy using adversarial regularization
Milad Nasr, Reza Shokri, and Amir Houmansadr · 2018
Earlier work this paper cites.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes · 2018
Earlier work this paper cites.
Privacy risk in machine learning: Analyzing the connection to overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha · 2018
Earlier work this paper cites.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song · 2019
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Earlier work this paper cites.
Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning
Milad Nasr, Reza Shokri, and Amir Houmansadr · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
White-box vs black-box: Bayes optimal strategies for membership inference
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou · 2019
Earlier work this paper cites.
Auditing data provenance in text-generation models
Congzheng Song and Vitaly Shmatikov · 2019
Earlier work this paper cites.
Demystifying membership inference attacks in machine learning as a service
Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Earlier work this paper cites.
The Pile: An 800gb dataset of diverse text for language modeling
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy · 2020
Earlier work this paper cites.
Auditing differentially private machine learning: How private is private sgd?
Matthew Jagielski, Jonathan Ullman, and Alina Oprea · 2020
Earlier work this paper cites.
Revisiting membership inference under realistic assumptions
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, and David Evans · 2020
Earlier work this paper cites.
Stolen memories: Leveraging model memorization for calibrated { \{ White-Box } \} membership inference
Klas Leino and Matt Fredrikson · 2020
Earlier work this paper cites.
A pragmatic approach to membership inferences on machine learning models
Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen · 2020
Earlier work this paper cites.
Information leakage in embedding models
Congzheng Song and Ananth Raghunathan · 2020
Earlier work this paper cites.
Understanding unintended memorization in federated learning
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays · 2020
Earlier work this paper cites.
Investigating the impact of pre-trained word embeddings on memorization in neural networks
Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow · 2020
Earlier work this paper cites.
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman · 2021
Earlier work this paper cites.
Extracting training data from large language models
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al · 2021
Earlier work this paper cites.
Label-only membership inference attacks
Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2021
Earlier work this paper cites.
Membership inference attacks against nlp classification models
Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim · 2021
Earlier work this paper cites.
On the importance of difficulty calibration in membership inference attacks
Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles · 2021
Earlier work this paper cites.
Finetuned language models are zero-shot learners
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le · 2021
Earlier work this paper cites.
What does it mean for a language model to preserve privacy?
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr · 2022
Earlier work this paper cites.
Membership inference attacks from first principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer · 2022
Earlier work this paper cites.
Quantifying memorization across neural language models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang · 2022
Earlier work this paper cites.
The privacy onion effect: Memorization is relative
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer · 2022
Cited alongside, same era.
Glam: Efficient scaling of language models with mixture-of-experts
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al · 2022
Cited alongside, same era.
Preventing verbatim memorization in language models gives a false sense of privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini · 2022
Cited alongside, same era.
Deduplicating training data mitigates privacy risks in language models
Nikhil Kandpal, Eric Wallace, and Colin Raffel · 2022
Cited alongside, same era.
Auditing membership leakages of multi-exit networks
Zheng Li, Yiyong Liu, Xinlei He, Ning Yu, Michael Backes, and Yang Zhang · 2022
Cited alongside, same era.
Tower: An open multilingual large language model for translation-related tasks, 2024
Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins · 2024
Closest in time.
Emergent and predictable memorization in large language models
Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff · 2024
Closest in time.
X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design, 2024
Eric L. Buehler and Markus J. Buehler · 2024
Closest in time.
Context-aware membership inference attacks against pre-trained large language models
Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Reza Shokri · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Membership inference attacks by exploiting loss trajectory
Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang · 2022
Cited alongside, same era.
Data contamination: From memorization to exploitation
Inbal Magar and Roy Schwartz · 2022
Cited alongside, same era.
An empirical analysis of memorization in fine-tuned autoregressive language models
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David K Evans, and Taylor Berg-Kirkpatrick · 2022
Cited alongside, same era.
Synthetic data–anonymisation groundhog day
Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso · 2022
Cited alongside, same era.
Enhanced membership inference attacks against machine learning models
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri · 2022
Cited alongside, same era.
Pythia: A suite for analyzing large language models across training and scaling, 2023
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal · 2023
Cited alongside, same era.
Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023
Together Computer · 2023
Cited alongside, same era.
Cyril Chhun, Fabian M Suchanek, and Chloé Clavel · 2024
Closest in time.
Saullm-7b: A pioneering large language model for law, 2024
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa · 2024
Closest in time.
Blind baselines beat membership inference attacks for foundation models
Debeshee Das, Jie Zhang, and Florian Tramèr · 2024
Closest in time.
Qlora: Efficient finetuning of quantized llms
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer · 2024
Closest in time.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li · 2024
Closest in time.
Do membership inference attacks work on large language models?
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi · 2024
Closest in time.
De-cop: Detecting copyrighted content in language models training data
André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li · 2024
Closest in time.
Cédric Eichler, Nathan Champeil, Nicolas Anciaux, Alexandra Bensamoun, Héber Hwang Arcolezi, and José Maria De Fuentes · 2024
Closest in time.
Croissantllm: A truly bilingual french-english language model
Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, António Loison, Duarte Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro Martins, et al · 2024
Closest in time.
Olmo: Accelerating the science of language models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi · 2024
Closest in time.
Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun · 2024
Closest in time.
Sampling-based pseudo-likelihood for membership inference attacks
Masahiro Kaneko, Youmi Ma, Yuki Wata, and Naoaki Okazaki · 2024
Closest in time.
Panoramia: Privacy auditing of machine learning models without retraining
Mishaal Kazmi, Hadrien Lautraite, Alireza Akbari, Mauricio Soroco, Qiaoyue Tang, Tao Wang, Sébastien Gambs, and Mathias Lécuyer · 2024
Closest in time.
Biomistral: A collection of open-source pretrained large language models for medical domains, 2024
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour · 2024
Closest in time.
Task contamination: Language models may not be few-shot anymore
Changmao Li and Jeffrey Flanigan · 2024
Closest in time.
Dora: Weight-decomposed low-rank adaptation, 2024
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen · 2024
Closest in time.
Probing language models for pre-training data detection
Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, and Wenliang Chen · 2024
Closest in time.
Llm dataset inference: Did you train on my dataset?
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic · 2024
Closest in time.
Did the neurons read your book? document-level membership inference for large language models
Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye · 2024
Closest in time.
Copyright traps for large language models
Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye · 2024
Closest in time.
Semantic membership inference attack against large language models
Hamid Mozaffari and Virendra J Marathe · 2024
Closest in time.
Can watermarking large language models prevent copyrighted text generation and hide training data?
Michael-Andrei Panaitescu-Liess, Zora Che, Bang An, Yuancheng Xu, Pankayaraj Pathmanathan, Souradip Chakraborty, Sicheng Zhu, Tom Goldstein, and Furong Huang · 2024
Closest in time.
Fast adversarial attacks on language models in one gpu minute
Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi · 2024
Closest in time.
Mosaic memory: Fuzzy duplication in copyright traps for large language models
Igor Shilov, Matthieu Meeus, and Yves-Alexandre de Montjoye · 2024
Closest in time.
Con-recall: Detecting pre-training data in llms via contrastive decoding
Cheng Wang, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, and Kai-Wei Chang · 2024
Closest in time.
Proving membership in llm pretraining data via data watermarks
Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia · 2024
Closest in time.
Recall: Membership inference via relative conditional log-likelihoods
Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra · 2024
Closest in time.
Data contamination can cross language barriers, 2024
Feng Yao, Yufan Zhuang, Zihao Sun, Sunan Xu, Animesh Kumar, and Jingbo Shang · 2024
Closest in time.
Machine unlearning of pre-trained large language models
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue · 2024
Closest in time.
Data contamination calibration for black-box llms
Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao · 2024
Closest in time.
Adaptive pre-training data detection for large language models via surprising tokens
Anqi Zhang and Chaofeng Wu · 2024
Closest in time.
Membership inference attacks cannot prove that a model was trained on your data
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr · 2024
Closest in time.
Min-k%++: Improved baseline for detecting pre-training data from large language models
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li · 2024
Closest in time.