Comparing large language models and human programmers for generating programming code

Understand

We systematically evaluated the performance of seven large language models in generating programming code using various prompt strategies, programming languages, and task difficulties.

• GPT-4 substantially outperforms other large language models, including Gemini Ultra and Claude 2.
• The coding performance of GPT-4 varies considerably with different prompt strategies.
• In most LeetCode and GeeksforGeeks coding contests evaluated in this study, GPT-4 employing the optimal prompt strategy outperforms 85 percent of human participants.