Fetching the paper…

Reading the bibliography…

The (ab)use of Open Source Code to Train Large Language Models · Around

2023

The (ab)use of Open Source Code to Train Large Language Models

Al-Kaswan, Ali, Izadi, Maliheh

Original
PDF
Post on X

Understand

In recent years, Large Language Models (LLMs) have gained significant popularity due to their ability to generate human-like text and their potential applications in various fields, such as Software Engineering.

LLMs for Code are commonly trained on large unsanitized corpora of source code scraped from the Internet.
The content of these datasets is memorized and emitted by the models, often in a verbatim manner.
In this work, we will discuss the security, privacy, and licensing implications of memorization.

Built on

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al. , “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21) , 2021, pp. 2633–2650
2021
Earlier work this paper cites.
M. Izadi, R. Gismondi, and G. Gousios, “Codefill: Multi-token code completion by jointly learning from structure and naming sequences,” in Proceedings of the 44th International Conference on Software Engineering (ICSE) . ACM, 2022, p. 401–412
2022
Earlier work this paper cites.

Similar

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in IEEE Symposium on Security and Privacy (SP) , 2022, pp. 754–768
2022
Cited alongside, same era.
Z. Sun, X. Du, F. Song, M. Ni, and L. Li, “Coprotector: Protect open-source code against unauthorized training usage with data poisoning,” in Proceedings of the ACM Web Conference 2022 , 2022, pp. 652–660
2022
Cited alongside, same era.

Then

A. Al-Kaswan, T. Ahmed, M. Izadi, A. A. Sawant, P. Devanbu, and A. van Deursen, “Extending source code pre-trained language models to summarise decompiled binaries,” in Proceedings of the 30th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2023
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…