2025

Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

Wang, Jianghui, Joshi, Vinay, Majumder, Saptarshi et al.

Understand

The demand for AI-generated GPU kernels is rapidly growing, influenced by the need for scalable, hardware-optimized solutions in both industry and academia.

  • As deep learning workloads grow in complexity and diversity, it is imperative to automate low-level kernel development to meet performance and productivity demands.
  • Major cloud providers, semiconductor companies, and research institutions are now investing heavily in AI-driven code generation for GPUs, aiming to reduce manual optimization efforts while achieving near-expert performance on hardware like AMD MI300X.
  • The Triton language, a Python-based DSL for GPU programming, has emerged as a popular target for such AI-generated kernels due to its balance of performance and ease-of-coding.

Built on

  • Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton · 2021

    Earlier work this paper cites.

  • Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang · 2023

    Earlier work this paper cites.

  • Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao · 2023

    Earlier work this paper cites.

  • Swe-bench: Can language models resolve real-world github issues?, 2024

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2024

    Earlier work this paper cites.

  • Learning performance-improving code edits, 2024

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh · 2024

    Earlier work this paper cites.

Similar

  • Can large language models write parallel code?

    Daniel Nichols, Joshua H. Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele · 2024

    Cited alongside, same era.

  • Hpc-coder-v2: Studying code llms across low-resource parallel languages, 2024

    Aman Chaturvedi, Daniel Nichols, Siddharth Singh, and Abhinav Bhatele · 2024

    Cited alongside, same era.

  • Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang · 2024

    Cited alongside, same era.

  • Qwen2.5-coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin · 2024

    Cited alongside, same era.

  • Large language models as optimizers, 2024

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen · 2024

    Cited alongside, same era.

Then

  • Tritonbench: Benchmarking large language model capabilities for generating triton operators, 2025

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, and Maosong Sun · 2025

    Closest in time.

  • Effibench: Benchmarking the efficiency of automatically generated code, 2025

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang · 2025

    Closest in time.

  • How efficient is llm-generated code? a rigorous & high-standard benchmark, 2025

    Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong · 2025

    Closest in time.

  • Kernelbench: Can llms write efficient gpu kernels?, 2025

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini · 2025

    Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…