Brevity is the soul of wit: Pruning long files for code generation

Understand

Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance.

• Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus.
• Specifically, many have shown that de-duplicating data, or sub-selecting higher quality data, can lead to efficiency or performance improvements.
• Generally, three types of methods are used to filter internet-scale corpora: embedding-based, heuristic-based, and classifier-based.