LLMs Provide Unstable Answers to Legal Questions

Statistics with Confidence—Confidence Intervals and Statistical Guidelines

M. J. Gardner and D. G. Altman. 1990 · 1990

Earlier work this paper cites.

274 Va. 657

Judicial Inquiry and Review Commission of Virginia v. Shull. 2007 · 2007

Earlier work this paper cites.

Randomization in Adjudication

Adam M. Samaha. 2009 · 2009

Earlier work this paper cites.

To trust or not to trust a classifier

Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018 · 2018

Earlier work this paper cites.

Addressing failure prediction by learning model confidence

Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. 2019 · 2019

Earlier work this paper cites.

Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering . 771–783

Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2020 · 2020

Earlier work this paper cites.

Language Models (Mostly) Know What They Know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, and et al. 2022 · 2022

Earlier work this paper cites.

Large Language Models are Zero-Shot Reasoners

Original

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Cited alongside, same era.

LLMediator: GPT-4 Assisted Online Dispute Resolution

Hannes Westermann, Jaromir Savelka, and Karim Benyekhlef. 2023 · 2023

Cited alongside, same era.

AI Tools for Legal Work: Claude, Gemini, Copilot, and More

American Bar Association. July 16, 2024 · 2024

Cited alongside, same era.

LLM Stability: A detailed analysis with some surprises

Original

Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024 · 2024

Cited alongside, same era.

Artificial intelligence tools for brief writing and analysis are a small firm litigator’s new best friend

Nicole Black. 2024 · 2024

Cited alongside, same era.

Large language models in law: A survey

Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S. Yu. 2024 · 2024

Later among the works it cites.

Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024 · 2024

Later among the works it cites.

Are Your LLMs Capable of Stable Reasoning?

Original

Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. 2024 · 2024

Later among the works it cites.

Caselaw Access Project

The President and Fellows of Harvard University. 2024 · 2024

Later among the works it cites.

The Effect of Sampling Temperature on Problem Solving in Large Language Models

Matthew Renze and Erhan Guven. 2024 · 2024

Later among the works it cites.