An Empirical Study of the Non-determinism of ChatGPT in Code Generation

Understand

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation.

• However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt.
• Non-determinism is a potential menace to scientific conclusion validity.
• When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses.