NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations

Understand

Large language models (LLMs) achieve promising results in code generation based on a given natural language description.

• They have been integrated into open-source projects and commercial products to facilitate daily coding activities.
• The natural language description in the prompt is crucial for LLMs to comprehend users' requirements.
• Prior studies uncover that LLMs are sensitive to the changes in the prompts, including slight changes that look inconspicuous.