RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” CoRR , vol. abs/2102.04664, 2021

Original

2021

Earlier work this paper cites.

L. Josipović, A. Guerrieri, and P. Ienne, “From c/c++ code to high-performance dataflow circuits,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 41, no. 7, pp. 2142–2155, 2022

2022

Earlier work this paper cites.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=iaYcJKpY2B{_}

2023

Earlier work this paper cites.

M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” 2023

2023

Earlier work this paper cites.

Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang, “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. [Online]. Available: https://openreview.net/forum?id=wgDcbBMSfh

2023

Earlier work this paper cites.

S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan-Gavitt, and S. Garg, “Benchmarking large language models for automated verilog rtl code generation,” in 2023 Design, Automation and Test in Europe Conference and Exhibition, DATE 2023 - Proceedings , ser. Proceedings -Design, Automation and Test in Europe, DATE. Institute of Electrical and Electronics Engineers Inc., 2023, publisher Copyright: © 2023 EDAA.; 2023 Design, Automation and Test in Europe Conference and Exhibition, DATE 2023 ; Conference date: 17-04-2023 Through 19-04-2023

2023

Cited alongside, same era.

T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue, “Opencodeinterpreter: Integrating code generation with execution and refinement,” https://arxiv.org/abs/2402.14658 , 2024

Original

2024

Cited alongside, same era.

M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu, B. Bhaskaran, B. Catanzaro, A. Chaudhuri, S. Clay, B. Dally, L. Dang, P. Deshpande, S. Dhodhi, S. Halepete, E. Hill, J. Hu, S. Jain, A. Jindal, B. Khailany, G. Kokai, K. Kunal, X. Li, C. Lind, H. Liu, S. Oberman, S. Omar, S. Pratty, J. Raiman, A. Sarkar, Z. Shao, H. Sun, P. P. Suthar, V. Tej, W. Turner, K. Xu, and H. Ren, “Chipnemo: Domain-adapted llms for chip design,” 2024

2024

Closest in time.

Y. Lu, S. Liu, Q. Zhang, and Z. Xie, “Rtllm: An open-source benchmark for design rtl generation with large language model,” in Proceedings of the 29th Asia and South Pacific Design Automation Conference , ser. ASPDAC ’24. IEEE Press, 2024, p. 722–727. [Online]. Available: https://doi.org/10.1109/ASP-DAC58780.2024.10473904

2024

Closest in time.

T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repository-level code auto-completion systems,” 2024. [Online]. Available: https://arxiv.org/abs/2306.03091

Original

2024

Closest in time.