ReCode: Robustness Evaluation of Code Generation Models

Understand

Code generation models have achieved impressive performance.

• However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood.
• Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation.
• In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models.