Language Models for Code Completion: A Practical Evaluation

Understand

Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data.

• This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code.
• We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models.
• We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions.