Fetching the paper…
Reading the bibliography…
Recent research has shown that language models have a tendency to memorize rare or unique token sequences in the training corpus.
“Calibrating noise to sensitivity in private data analysis,”
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith, · 2006
Earlier work this paper cites.
“The EU proposal for a general data protection regulation and the roots of the ‘right to be forgotten’,”
Alessandro Mantelero, · 2013
Earlier work this paper cites.
“The algorithmic foundations of differential privacy,”
Cynthia Dwork and Aaron Roth, · 2014
Earlier work this paper cites.
“Model inversion attacks that exploit confidence information and basic countermeasures,”
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart, · 2015
Earlier work this paper cites.
“LibriSpeech: an ASR corpus based on public domain audio books,”
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, · 2015
Earlier work this paper cites.
“Federated learning: Strategies for improving communication efficiency,”
Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon, · 2016
Earlier work this paper cites.
“Pointer sentinel mixture models,”
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher, · 2016
Earlier work this paper cites.
“Communication-efficient learning of deep networks from decentralized data,”
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas, · 2017
Earlier work this paper cites.
“Semi-supervised knowledge transfer for deep learning from private training data,”
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar, · 2017
Earlier work this paper cites.
“An analysis of incorporating an external language model into a sequence-to-sequence model,”
Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar, · 2018
Earlier work this paper cites.
“Scalable private learning with PATE,”
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson, · 2018
Cited alongside, same era.
“Auditing data provenance in text-generation models,”
Congzheng Song and Vitaly Shmatikov, · 2019
Cited alongside, same era.
“The secret sharer: Evaluating and testing unintended memorization in neural networks,”
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song, · 2019
Cited alongside, same era.
“Language modeling with deep transformers,”
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney, · 2019
Cited alongside, same era.
“Differential privacy has disparate impact on model accuracy,”
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov, · 2019
Cited alongside, same era.
“Enhance PATE on complex tasks with knowledge transferred from non-private data,”
Lulu Wang, Junxiang Zheng, Yongzhi Cao, and Hanpin Wang, · 2019
“Machine unlearning,”
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot, · 2021
Later among the works it cites.
“Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,”
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer, · 2021
Later among the works it cites.
“LoRA: Low-rank adaptation of large language models,”
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, · 2021
Later among the works it cites.
“Are large pre-trained language models leaking your personal information?,”
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang, · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
“Training production language models without memorizing user data,”
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays, · 2020
Cited alongside, same era.
“Understanding unintended memorization in language models under federated learning,”
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays, · 2020
Cited alongside, same era.
Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu, · 2020
Cited alongside, same era.
“Extracting training data from large language models,”
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al., · 2021
Cited alongside, same era.
“LibriSpeech language models, vocabulary and G2P models,” https://www.openslr.org/11/
Cited in the paper.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, · 2022
Later among the works it cites.
“Detecting unintended memorization in language-model-fused ASR,”
W Ronny Huang, Steve Chien, Om Thakkar, and Rajiv Mathews, · 2022
Later among the works it cites.
“Measuring forgetting of memorized training examples,”
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al., · 2022
Later among the works it cites.
“Knowledge unlearning for mitigating privacy risks in language models,”
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo, · 2022
Later among the works it cites.
“Mitigating unintended memorization in language models via alternating teaching,”
Zhe Liu, Xuedong Zhang, and Fuchun Peng, · 2023
Closest in time.
“Llama 2: Open foundation and fine-tuned chat models,”
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al., · 2023
Closest in time.