The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers

Understand

Evaluation of large language models for code has primarily relied on static benchmarks, including HumanEval (Chen et al., 2021), or more recently using human preferences of LLM responses.

• As LLMs are increasingly used as programmer assistants, we study whether gains on existing benchmarks or more preferred LLM responses translate to programmer productivity when coding with LLMs, including time spent coding.
• We introduce RealHumanEval, a web interface to measure the ability of LLMs to assist programmers, through either autocomplete or chat support.
• We conducted a user study (N=243) using RealHumanEval in which users interacted with seven LLMs of varying base model performance.