Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement

Understand

Generative language models (LMs) are increasingly used for document class-prediction tasks and promise enormous improvements in cost and efficiency.

• Existing research often examines simple classification tasks, but the capability of LMs to classify on complex or specialized tasks is less well understood.
• We consider a highly complex task that is challenging even for humans: the classification of legal reasoning according to jurisprudential philosophy.
• Using a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts, we systematically test the performance of a variety of LMs.