Built on

B. Danet, “Language in the legal process,” Law & Society Review , vol. 14, no. 3, pp. 445–564, 1980
1980
Earlier work this paper cites.
A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
A. Radford, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Earlier work this paper cites.
I. Chalkidis, A. Jana, D. Hartung, M. Bommarito, I. Androutsopoulos, D. M. Katz, and N. Aletras, “Lexglue: A benchmark dataset for legal language understanding in english,” arXiv preprint arXiv:2110.00976 , 2021
Original
2021
Earlier work this paper cites.
M. J. Bommarito II, D. M. Katz, and E. M. Detterman, “Lexnlp: Natural language processing and information extraction for legal and regulatory texts,” in Research handbook on big data law . Edward Elgar Publishing, 2021, pp. 216–227
2021
Earlier work this paper cites.
J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y. Liu, S. Zhang et al. , “Review of large vision models and visual prompt engineering,” Meta-Radiology , p. 100047, 2023
2023
Earlier work this paper cites.
I. Chalkidis, “Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark,” arXiv preprint arXiv:2304.12202 , 2023
Original
2023
Earlier work this paper cites.
L. Hongcheng, L. Yusheng, M. Yutong, and Y. Wang, “Xiezhi: Chinese law large language model,” https://github.com/LiuHC0428/LAW_GPT
2023
Earlier work this paper cites.
P. P. Ray, “Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet of Things and Cyber-Physical Systems , vol. 3, pp. 121–154, 2023
2023
Earlier work this paper cites.
G.-G. Lee, L. Shi, E. Latif, Y. Gao, A. Bewersdorff, M. Nyaaba, S. Guo, Z. Wu, Z. Liu, H. Wang et al. , “Multimodality of ai for education: Towards artificial general intelligence,” arXiv preprint arXiv:2312.06037 , 2023
Original
2023
Earlier work this paper cites.
A. Deroy, K. Ghosh, and S. Ghosh, “How ready are pre-trained abstractive models and llms for legal case judgement summarization?” arXiv preprint arXiv:2306.01248 , 2023
Original
2023
Earlier work this paper cites.
J. Savelka, K. D. Ashley, M. A. Gray, H. Westermann, and H. Xu, “Explaining legal concepts with augmented large language models (gpt-4),” arXiv preprint arXiv:2306.09525 , 2023
Original
2023
Earlier work this paper cites.
A. Simmons and R. Vasa, “Garbage in, garbage out: Zero-shot detection of crime using large language models,” arXiv preprint arXiv:2307.06844 , 2023
Original
2023
Earlier work this paper cites.
Q. Huang, M. Tao, C. Zhang, Z. An, C. Jiang, Z. Chen, Z. Wu, and Y. Feng, “Lawyer llama technical report,” 2023
2023
Earlier work this paper cites.
——, “Lawyer llama,” https://github.com/AndrewZhe/lawyer-llama
2023
Earlier work this paper cites.
J.-S. Lee, “Lexgpt 0.1: pre-trained gpt-j models with pile of law,” arXiv preprint arXiv:2306.05431 , 2023
Original
2023
Earlier work this paper cites.
J. Cui, Z. Li, Y. Yan, B. Chen, and L. Yuan, “Chatlaw: Open-source legal large language model with integrated external knowledge bases,” arXiv preprint arXiv:2306.16092 , 2023
Original
2023
Earlier work this paper cites.
S. Yue, W. Chen, S. Wang, B. Li, C. Shen, S. Liu, Y. Zhou, Y. Xiao, S. Yun, X. Huang et al. , “Disc-lawllm: Fine-tuning large language models for intelligent legal services,” arXiv preprint arXiv:2309.11325 , 2023
Original
2023
Earlier work this paper cites.
W. Deng, J. Pei, K. Kong, Z. Chen, F. Wei, Y. Li, Z. Ren, Z. Chen, and P. Ren, “Syllogistic reasoning for legal judgment analysis,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 13 997–14 009. [Online]. Available: https://aclanthology.org/2023.emnlp-main.864
2023
Earlier work this paper cites.

Similar

Then

L. M. D. Droit, “Docket alarm incorporates gpt-3.5 to auto-summarize pdf litigation filings in complex dockets,” 2023, accessed: 2024-10-27. [Online]. Available: https://www.lemondedudroit.fr/professions/337-legaltech/85951-chatgpt-predictice-integremoteur-recherche.html
2024
Closest in time.
T. Hassonjee, “Navigating ai for legal documents: Tips & tricks,” 2024, accessed: 2024-10-27. [Online]. Available: https://www.docdraft.ai/blogs/navigating-ai-for-legal-documents-tips-tricks
2024
Closest in time.
Henchman, “Lexisnexis to acquire henchman: A new chapter in legal drafting,” 2024, accessed: 2024-10-27. [Online]. Available: https://henchman.io/blog/lexisnexis-announcement
2024
Closest in time.
G. Lambert, “Colin lachance on jurisage’s myjr and how he’s looking at ai to assist in the synthesis and reading of legal cases,” 2023, accessed: 2024-10-27. [Online]. Available: https://tinyurl.com/4tnd29uv
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Legal Evalutions and Challenges of Large Language Models

Wang, Jiaqi, Zhao, Huan, Yang, Zhenyuan et al.

Understand

In this paper, we review legal testing methods based on Large Language Models (LLMs), using the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions.

We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain.
Systematic tests are conducted on English and Chinese legal cases, and the results are analyzed in depth.
Through systematic testing of legal cases from common law systems and China, this paper explores the strengths and weaknesses of LLMs in understanding and applying legal texts, reasoning through legal issues, and predicting judgments.

Built on

B. Danet, “Language in the legal process,” Law & Society Review , vol. 14, no. 3, pp. 445–564, 1980
1980
Earlier work this paper cites.
A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
A. Radford, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Earlier work this paper cites.
I. Chalkidis, A. Jana, D. Hartung, M. Bommarito, I. Androutsopoulos, D. M. Katz, and N. Aletras, “Lexglue: A benchmark dataset for legal language understanding in english,” arXiv preprint arXiv:2110.00976 , 2021
Original
2021
Earlier work this paper cites.
M. J. Bommarito II, D. M. Katz, and E. M. Detterman, “Lexnlp: Natural language processing and information extraction for legal and regulatory texts,” in Research handbook on big data law . Edward Elgar Publishing, 2021, pp. 216–227
2021
Earlier work this paper cites.
J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y. Liu, S. Zhang et al. , “Review of large vision models and visual prompt engineering,” Meta-Radiology , p. 100047, 2023
2023
Earlier work this paper cites.
I. Chalkidis, “Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark,” arXiv preprint arXiv:2304.12202 , 2023
Original
2023
Earlier work this paper cites.
L. Hongcheng, L. Yusheng, M. Yutong, and Y. Wang, “Xiezhi: Chinese law large language model,” https://github.com/LiuHC0428/LAW_GPT
2023
Earlier work this paper cites.
P. P. Ray, “Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet of Things and Cyber-Physical Systems , vol. 3, pp. 121–154, 2023
2023
Earlier work this paper cites.
G.-G. Lee, L. Shi, E. Latif, Y. Gao, A. Bewersdorff, M. Nyaaba, S. Guo, Z. Wu, Z. Liu, H. Wang et al. , “Multimodality of ai for education: Towards artificial general intelligence,” arXiv preprint arXiv:2312.06037 , 2023
Original
2023
Earlier work this paper cites.
A. Deroy, K. Ghosh, and S. Ghosh, “How ready are pre-trained abstractive models and llms for legal case judgement summarization?” arXiv preprint arXiv:2306.01248 , 2023
Original
2023
Earlier work this paper cites.
J. Savelka, K. D. Ashley, M. A. Gray, H. Westermann, and H. Xu, “Explaining legal concepts with augmented large language models (gpt-4),” arXiv preprint arXiv:2306.09525 , 2023
Original
2023
Earlier work this paper cites.
A. Simmons and R. Vasa, “Garbage in, garbage out: Zero-shot detection of crime using large language models,” arXiv preprint arXiv:2307.06844 , 2023
Original
2023
Earlier work this paper cites.
Q. Huang, M. Tao, C. Zhang, Z. An, C. Jiang, Z. Chen, Z. Wu, and Y. Feng, “Lawyer llama technical report,” 2023
2023
Earlier work this paper cites.
——, “Lawyer llama,” https://github.com/AndrewZhe/lawyer-llama
2023
Earlier work this paper cites.
J.-S. Lee, “Lexgpt 0.1: pre-trained gpt-j models with pile of law,” arXiv preprint arXiv:2306.05431 , 2023
Original
2023
Earlier work this paper cites.
J. Cui, Z. Li, Y. Yan, B. Chen, and L. Yuan, “Chatlaw: Open-source legal large language model with integrated external knowledge bases,” arXiv preprint arXiv:2306.16092 , 2023
Original
2023
Earlier work this paper cites.
S. Yue, W. Chen, S. Wang, B. Li, C. Shen, S. Liu, Y. Zhou, Y. Xiao, S. Yun, X. Huang et al. , “Disc-lawllm: Fine-tuning large language models for intelligent legal services,” arXiv preprint arXiv:2309.11325 , 2023
Original
2023
Earlier work this paper cites.
W. Deng, J. Pei, K. Kong, Z. Chen, F. Wei, Y. Li, Z. Ren, Z. Chen, and P. Ren, “Syllogistic reasoning for legal judgment analysis,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 13 997–14 009. [Online]. Available: https://aclanthology.org/2023.emnlp-main.864
2023
Earlier work this paper cites.

Similar

Then

L. M. D. Droit, “Docket alarm incorporates gpt-3.5 to auto-summarize pdf litigation filings in complex dockets,” 2023, accessed: 2024-10-27. [Online]. Available: https://www.lemondedudroit.fr/professions/337-legaltech/85951-chatgpt-predictice-integremoteur-recherche.html
2024
Closest in time.
T. Hassonjee, “Navigating ai for legal documents: Tips & tricks,” 2024, accessed: 2024-10-27. [Online]. Available: https://www.docdraft.ai/blogs/navigating-ai-for-legal-documents-tips-tricks
2024
Closest in time.
Henchman, “Lexisnexis to acquire henchman: A new chapter in legal drafting,” 2024, accessed: 2024-10-27. [Online]. Available: https://henchman.io/blog/lexisnexis-announcement
2024
Closest in time.
G. Lambert, “Colin lachance on jurisage’s myjr and how he’s looking at ai to assist in the synthesis and reading of legal cases,” 2023, accessed: 2024-10-27. [Online]. Available: https://tinyurl.com/4tnd29uv
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J. Lai, W. Gan, J. Wu, Z. Qi, and S. Y. Philip, “Large language models in law: A survey,” AI Open , 2024

2024

Cited alongside, same era.

B. Chaudhary, P. Covarrubia, and G. Y. Ng, “The judge, the ai, and the crown: a collusive network,” Information & Communications Technology Law , vol. 33, no. 3, pp. 330–367, 2024

2024

Cited alongside, same era.

Z. Zhou, J.-X. Shi, P.-X. Song, X.-W. Yang, Y.-X. Jin, L.-Z. Guo, and Y.-F. Li, “Lawgpt: A chinese legal knowledge-enhanced large language model,” arXiv preprint arXiv:2406.04614 , 2024

Original

2024

Cited alongside, same era.

N. Guha, J. Nyarko, D. Ho, C. Ré, A. Chilton, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano et al. , “Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Cited alongside, same era.

M. U. Hadi, Q. Al Tashi, A. Shah, R. Qureshi, A. Muneer, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu et al. , “Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects,” Authorea Preprints , 2024

2024

Cited alongside, same era.

I. Cheong, A. Caliskan, and T. Kohno, “Safeguarding human values: rethinking us law for generative ai’s societal impacts,” AI and Ethics , pp. 1–27, 2024

2024

Cited alongside, same era.

F. Osasona, O. O. Amoo, A. Atadoga, T. O. Abrahams, O. A. Farayola, and B. S. Ayinla, “Reviewing the ethical implications of ai in decision making processes,” International Journal of Management & Entrepreneurship Research , vol. 6, no. 2, pp. 322–335, 2024

2024

Cited alongside, same era.

C. U. Akpuokwe, A. O. Adeniyi, and S. S. Bakare, “Legal challenges of artificial intelligence and robotics: a comprehensive review,” Computer Science & IT Research Journal , vol. 5, no. 3, pp. 544–561, 2024

2024

Cited alongside, same era.

B. J, “Ask blue j enhances user experience through gpt-4 and conversational,” 2023, accessed: 2024-10-27. [Online]. Available: https://www.bluej.com/blog/ask-blue-j-enhances-user-experience

2024

Closest in time.

LexiLaw, “Lexilaw: a chinese legal large language model,” 2023, accessed: 2024-10-27. [Online]. Available: https://github.com/CSHaitao/LexiLaw?tab=readme-ov-file#readme

2024

Closest in time.

KL3M, “Meet kl3m: the first legal large language model,” 2024, accessed: 2024-10-27. [Online]. Available: https://273ventures.com/kl3m-the-first-legal-large-language-model/#note1

2024

Closest in time.

Z. Fei, S. Zhang, X. Shen, D. Zhu, X. Wang, M. Cao, F. Zhou, Y. Li, W. Zhang, D. Lin et al. , “Internlm-law: An open source chinese legal large language model,” arXiv preprint arXiv:2406.14887 , 2024

Original

2024

Closest in time.

P. Colombo, T. Pires, M. Boudiaf, R. Melo, D. Culver, S. Morgado, E. Malaboeuf, G. Hautreux, J. Charpentier, and M. Desa, “Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain,” arXiv preprint arXiv:2407.19584 , 2024

Original

2024

Closest in time.

Y. Wu, Y. Liu, Y. Liu, A. Li, S. Zhou, and K. Kuang, “wisdominterrogatory,” 2024, accessed: 2024-10-27. [Online]. Available: https://github.com/zhihaiLLM/wisdomInterrogatory

2024

Closest in time.

G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

Original

2024

Closest in time.

A. Anthropic, “Claude 3.5 sonnet model card addendum,” Claude-3.5 Model Card , 2024

2024

Closest in time.

A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang et al. , “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652 , 2024

Original

2024