Generative Data Refinement: Just Ask for Better Data

Understand

For a fixed parameter size, the capabilities of large models are primarily determined by the quality and quantity of its training data.

• Consequently, training datasets now grow faster than the rate at which new data is indexed on the web, leading to projected data exhaustion over the next decade.
• Much more data exists as user-generated content that is not publicly indexed, but incorporating such data comes with considerable risks, such as leaking private information and other undesirable content.
• We introduce a framework, Generative Data Refinement (GDR), for using pretrained generative models to transform a dataset with undesirable content into a refined dataset that is more suitable for training.