VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation

G. Huang, J. Hu, Y. He, J. Liu, M. Ma, Z. Shen, J. Wu, Y. Xu, H. Zhang, K. Zhong et al. , “Machine learning for electronic design automation: A survey,” ACM Transactions on Design Automation of Electronic Systems (TODAES) , vol. 26, no. 5, pp. 1–46, 2021

2021

Earlier work this paper cites.

M. Rapp, H. Amrouch, Y. Lin, B. Yu, D. Z. Pan, M. Wolf, and J. Henkel, “MLCAD: A survey of research in machine learning for CAD keynote paper,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 41, no. 10, pp. 3162–3181, 2021

2021

Earlier work this paper cites.

M. Chen and et al., “Evaluating large language models trained on code,” 2021

2021

Earlier work this paper cites.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

Original

2022

Earlier work this paper cites.

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, “Peft: State-of-the-art parameter-efficient fine-tuning methods,” Younes Belkada and Sayak Paul,” PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods , 2022

2022

Earlier work this paper cites.

S. Thakur, J. Blocklove, H. Pearce, B. Tan, S. Garg, and R. Karri, “Autochip: Automating HDL generation using LLM feedback,” arXiv preprint arXiv:2311.04887 , 2023

Original

2023

Earlier work this paper cites.

R. Zhong, X. Du, S. Kai, Z. Tang, S. Xu, H.-L. Zhen, J. Hao, Q. Xu, M. Yuan, and J. Yan, “LLM4EDA: Emerging progress in large language models for electronic design automation,” arXiv preprint arXiv:2401.12224 , 2023

Original

2023

Earlier work this paper cites.

S. Liu, W. Fang, Y. Lu, Q. Zhang, H. Zhang, and Z. Xie, “RTLCoder: Outperforming GPT-3.5 in design RTL generation with our open-source dataset and lightweight solution,” arXiv preprint arXiv:2312.08617 , 2023

Original

2023

Earlier work this paper cites.

K. Chang, Y. Wang, H. Ren, M. Wang, S. Liang, Y. Han, H. Li, and X. Li, “Chipgpt: How far are we from natural language hardware design,” arXiv preprint arXiv:2305.14019 , 2023

Original

2023

Cited alongside, same era.

J. Blocklove, S. Garg, R. Karri, and H. Pearce, “Chip-chat: Challenges and opportunities in conversational hardware design,” in 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD) . IEEE, 2023, pp. 1–6

2023

Cited alongside, same era.

M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu et al. , “ChipNeMo: Domain-adapted LLMs for chip design,” arXiv preprint arXiv:2311.00176 , 2023

Original

2023

Cited alongside, same era.

M. Nair, R. Sadhukhan, and D. Mukhopadhyay, “Generating secure hardware using chatgpt resistant to cwes,” Cryptology ePrint Archive , 2023

2023

Cited alongside, same era.

E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, “Codegen2: Lessons for training LLMs on programming and natural languages,” arXiv preprint arXiv:2305.02309 , 2023

Original

2023

Later among the works it cites.

M. Liu, N. Pinckney, B. Khailany, and H. Ren, “VerilogEval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) . IEEE, 2023, pp. 1–8

2023

Later among the works it cites.

K. Zhou, Y. Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y. Lin, J.-R. Wen, and J. Han, “Don’t make your LLM an evaluation benchmark cheater,” arXiv preprint arXiv:2311.01964 , 2023

Original

2023

Later among the works it cites.

T. Dettmers and L. Zettlemoyer, “The case for 4-bit precision: k-bit inference scaling laws,” in International Conference on Machine Learning . PMLR, 2023, pp. 7750–7774

2023

Later among the works it cites.