SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore

Understand

The legality of training language models (LMs) on copyrighted or otherwise restricted data is under intense debate.

• However, as we show, model performance significantly degrades if trained only on low-risk text (e.g., out-of-copyright books or government documents), due to its limited size and domain coverage.
• We present SILO, a new language model that manages this risk-performance tradeoff during inference.
• SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with a more general and easily modifiable nonparametric datastore (e.g., containing copyrighted books or news) that is only queried during inference.