User Centric Evaluation of Code Generation Tools

Understand

With the rapid advance of machine learning (ML) technology, large language models (LLMs) are increasingly explored as an intelligent tool to generate program code from natural language specifications.

• However, existing evaluations of LLMs have focused on their capabilities in comparison with humans.
• It is desirable to evaluate their usability when deciding on whether to use a LLM in software production.
• This paper proposes a user centric method for this purpose.