Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

Understand

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information.

• Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc and failed to take context into account.
• We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material.
• First, we gather and make available the Pile of Law, a 256GB (and growing) dataset of open-source English-language legal and administrative data, covering court opinions, contracts, administrative rules, and legislative records.