Execution-Based Evaluation for Open-Domain Code Generation

Understand

To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset.

• ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution.
• Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries.
• Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian.