Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness

F. Mahdisoltani, J. Biega, and F. M. Suchanek, “Yago3: A knowledge base from multilingual wikipedias,” in CIDR

2013

Earlier work this paper cites.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning

2017

Earlier work this paper cites.

T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, “Convolutional 2d knowledge graph embeddings,” in Proceedings of the AAAI conference on artificial intelligence

2018

Earlier work this paper cites.

F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?,” arXiv preprint arXiv:1909.01066

Original

2019

Earlier work this paper cites.

A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2020

Earlier work this paper cites.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300

Original

2020

Earlier work this paper cites.

C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, “Certified data removal from machine learning models,” in Proceedings of the 37th International Conference on Machine Learning

2020

Earlier work this paper cites.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al

Original

2021

Earlier work this paper cites.

D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, “Knowledge neurons in pretrained transformers,” arXiv preprint arXiv:2104.08696

Original

2021

Earlier work this paper cites.

N. De Cao, W. Aziz, and I. Titov, “Editing factual knowledge in language models,” arXiv preprint arXiv:2104.08164

Original

2021

Earlier work this paper cites.

L. Graves, V. Nagisetty, and V. Ganesh, “Amnesiac machine learning,” in Proceedings of the AAAI Conference on Artificial Intelligence

2021

Earlier work this paper cites.

X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, “Quark: Controllable text generation with reinforced unlearning,” Advances in neural information processing systems

2022

Earlier work this paper cites.

M. Zamini, H. Reza, and M. Rabiei, “A review of knowledge graph completion,” Information

2022

Earlier work this paper cites.

J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo, “Knowledge unlearning for mitigating privacy risks in language models,” arXiv preprint arXiv:2210.01504

Original

2022

Earlier work this paper cites.

V. B. Kumar, R. Gangadharaiah, and D. Roth, “Privacy adhering machine un-learning in nlp,” arXiv preprint arXiv:2212.09573

Original

2022

Earlier work this paper cites.

K. Meng, D. Bau, A. Andonian, and Y. Belinkov, “Locating and editing factual associations in gpt,” Advances in neural information processing systems

2022

Earlier work this paper cites.

K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, “Mass-editing memory in a transformer,” arXiv preprint arXiv:2210.07229

Original

2022

Earlier work this paper cites.

Q. Dong, D. Dai, Y. Song, J. Xu, Z. Sui, and L. Li, “Calibrating factual knowledge in pretrained language models,” arXiv preprint arXiv:2210.03329

Original

2022

Earlier work this paper cites.

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al

Original

2022

Earlier work this paper cites.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al

2022

Earlier work this paper cites.

H. H.-H. Hsu, Y. Shen, C. Tomani, and D. Cremers, “What makes graph neural networks miscalibrated?,” Advances in Neural Information Processing Systems

2022

Earlier work this paper cites.

J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, N. G. Laleh, C. M. L. Löffler, S.-C. Schwarzkopf, M. Unger, G. P. Veldhuizen, et al

2023

Cited alongside, same era.

K. K. Chang, M. Cramer, S. Soni, and D. Bamman, “Speak, memory: An archaeology of books known to chatgpt/gpt-4,” arXiv preprint arXiv:2305.00118

Original

2023

Cited alongside, same era.

J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” arXiv preprint arXiv:2310.20150

Original

2023

Cited alongside, same era.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al

Original

2023

Cited alongside, same era.

Z. Zhong, Z. Wu, C. D. Manning, C. Potts, and D. Chen, “Mquake: Assessing knowledge editing in language models via multi-hop questions,” arXiv preprint arXiv:2305.14795

R. Wei, M. Li, M. Ghassemi, E. Kreačić, Y. Li, X. Yue, B. Li, V. K. Potluru, P. Li, and E. Chien, “Underestimated privacy risks for minority populations in large language model unlearning,” arXiv preprint arXiv:2412.08559

Original

2024

Later among the works it cites.

W. Ma, X. Feng, W. Zhong, L. Huang, Y. Ye, X. Feng, and B. Qin, “Unveiling entity-level unlearning for large language models: A comprehensive analysis,” arXiv preprint arXiv:2406.15796

Original

2024

Later among the works it cites.

Y. Shi, Q. Tan, X. Wu, S. Zhong, K. Zhou, and N. Liu, “Retrieval-enhanced knowledge editing in language models for multi-hop question answering,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management

2024

Later among the works it cites.

B. Wei, W. Shi, Y. Huang, N. A. Smith, C. Zhang, L. Zettlemoyer, K. Li, and P. Henderson, “Evaluating copyright takedown methods for language models,” arXiv preprint arXiv:2406.18664

Original

2024

Later among the works it cites.