Toxicity of the Commons: Curating Open-Source Pre-Training Data

Understand

Open-source large language models are becoming increasingly available and popular among researchers and practitioners.

• While significant progress has been made on open-weight models, open training data is a practice yet to be adopted by the leading open-weight models creators.
• At the same time, there researchers are working to make language models safer.
• We propose a data curation pipeline to reduce harmful outputs by models trained on public domain data.