LiCoEval: Evaluating LLMs on License Compliance in Code Generation

U. C. of Appeals for the Second Circuit. (1946, February) Arnstein v. porter, 154 f.2d 464 (2d cir. 1946). [Online]. Available: https://law.justia.com/cases/federal/appellate-courts/F2/154/464/1478575/

1946

Earlier work this paper cites.

V. I. Levenshtein et al. , “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady , vol. 10, no. 8. Soviet Union, 1966, pp. 707–710

1966

Earlier work this paper cites.

T. J. McCabe, “A complexity measure,” IEEE Transactions on software Engineering , no. 4, pp. 308–320, 1976

1976

Earlier work this paper cites.

N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.” Psychological bulletin , vol. 114, no. 3, p. 494, 1993

1993

Earlier work this paper cites.

A. Z. Broder, “Identifying and filtering near-duplicate documents,” in Annual symposium on combinatorial pattern matching . Springer, 2000, pp. 1–10

2000

Earlier work this paper cites.

J. R. Autry, “Toward a definition of striking similarity in infringement actions for copyrighted musical works,” J. Intell. Prop. L. , vol. 10, p. 113, 2002

2002

Earlier work this paper cites.

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

2002

Earlier work this paper cites.

S. R. Higgins, “Proving copyright infringement: Will striking similarity make your case,” Suffolk J. Trial & App. Advoc. , vol. 8, p. 157, 2003

2003

Earlier work this paper cites.

L. Rosen, “Open source licensing,” Software Freedom and Intellectual Property Law , 2005

2005

Earlier work this paper cites.

D. M. German and A. E. Hassan, “License integration patterns: Addressing license mismatches in component-based development,” in 2009 IEEE 31st international conference on software engineering . IEEE, 2009, pp. 188–198

2009

Earlier work this paper cites.

P. E. McKnight and J. Najab, “Mann-whitney u test,” The Corsini encyclopedia of psychology , pp. 1–1, 2010

2010

Earlier work this paper cites.

J. Reddy. (2015, Augst) The consequences of violating open source licenses. [Online]. Available: https://btlj.org/2015/11/consequences-violating-open-source-licenses/

2015

Earlier work this paper cites.

K. Srinath, “Python–the fastest growing programming language,” International Research Journal of Engineering and Technology , vol. 4, no. 12, pp. 354–357, 2017

2017

Earlier work this paper cites.

Y. Ma, C. Bogart, S. Amreen, R. Zaretzki, and A. Mockus, “World of code: an infrastructure for mining the universe of open source vcs data,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) . IEEE, 2019, pp. 143–154

2019

Earlier work this paper cites.

H. J. Meeker, Open (Source) for Business: A Practical Guide to Open Source Software Licensing - Third Edition . United States: Independently Published, 2020

2020

Earlier work this paper cites.

S. Karpinski. (2021, July) I don’t want to say anything but that’s not the right license mr copilot. [Online]. Available: https://twitter.com/stefankarpinski/status/1410971061181681674

2021

Earlier work this paper cites.

Eevee. (2021, June) I’m unclear on how it’s not a form of laundering open source code into commercial works. [Online]. Available: https://twitter.com/eevee/status/1410037309848752128

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

M. V. Pour, Z. Li, L. Ma, and H. Hemmati, “A search-based testing framework for deep neural networks of source code embedding,” in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) . IEEE, 2021, pp. 36–46

2021

Earlier work this paper cites.

C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini, “Counterfactual memorization in neural language models,” arXiv preprint arXiv:2112.12938 , 2021

Original

2021

Earlier work this paper cites.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

Original

2021

Earlier work this paper cites.

R. Aleithan, “Explainable just-in-time bug prediction: Are we there yet?” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 2021, pp. 129–131

2021

Earlier work this paper cites.

W. Xu and X. Zhang, “Multi-granularity code smell detection using deep learning method based on abstract syntax tree.” in SEKE , 2021, pp. 503–509

2021

Earlier work this paper cites.

D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999 , 2022

Original

2022

Earlier work this paper cites.

M. Izadi, R. Gismondi, and G. Gousios, “Codefill: Multi-token code completion by jointly learning from structure and naming sequences,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 401–412

2022

Earlier work this paper cites.

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 754–768

2022

Earlier work this paper cites.

T. Davis. (2022, October) copilot, with ”public code” blocked, emits large chunks of my copyrighted code, with no attribution, no lgpl license. [Online]. Available: https://twitter.com/DocSparse/status/1581461734665367554

2022

Earlier work this paper cites.

C. Green. (2022, June) Explored github copilot,a paid service, to see if it encodes code from repositories with restrictive licenses. [Online]. Available: https://twitter.com/ChrisGr93091552/status/1539731632931803137

2022

Earlier work this paper cites.

Github. (2022, August) Github copilot. [Online]. Available: https://github.com/features/copilot

2022

Earlier work this paper cites.

(2022, November) Github copilot litigation. [Online]. Available: https://githubcopilotlitigation.com/

2022

Earlier work this paper cites.

Z. Yang, J. Shi, J. He, and D. Lo, “Natural attack for pre-trained models of code,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1482–1493

2022

Earlier work this paper cites.

U. S. C. for the Ninth Circuit. (2022, December) Copying—access and substantial similarity. [Online]. Available: https://www.ce9.uscourts.gov/jury-instructions/node/274

2022

Cited alongside, same era.

S. Scheffler, E. Tromer, and M. Varia, “Formalizing human ingenuity: A quantitative framework for copyright law’s substantial similarity,” in Proceedings of the 2022 Symposium on Computer Science and Law , 2022, pp. 37–49

2022

Cited alongside, same era.

N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646 , 2022

Original

2022

Cited alongside, same era.

K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating training data makes language models better,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 8424–8445. [Online]. Available: https://aclanthology.org/2022.acl-long.577

A. Al-Kaswan, M. Izadi, and A. V. Deursen, “Traces of memorisation in large language models for code,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) , 2024, pp. 862–862

2024

Closest in time.

(2024, March) The Apache 2.0 license. [Online]. Available: https://www.apache.org/licenses/LICENSE-2.0

2024

Closest in time.

H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

2024

Closest in time.

J. Wu, L. Bao, X. Yang, X. Xia, and X. Hu, “A large-scale empirical study of open source license usage: Practices and challenges,” in 2024 IEEE/ACM 21th International Conference on Mining Software Repositories (MSR) , 2024

2024

Closest in time.