2022

Legal Prompt Engineering for Multilingual Legal Judgement Prediction

Trautmann, Dietrich, Petrova, Alina, Schilder, Frank

Understand

Legal Prompt Engineering (LPE) or Legal Prompting is a process to guide and assist a large language model (LLM) with performing a natural legal language processing (NLLP) skill.

Our goal is to use LPE with LLMs over long legal documents for the Legal Judgement Prediction (LJP) task.
We investigate the performance of zero-shot LPE for given facts in case-texts from the European Court of Human Rights (in English) and the Federal Supreme Court of Switzerland (in German, French and Italian).
Our results show that zero-shot LPE is better compared to the baselines, but it still falls short compared to current state of the art supervised approaches.

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Machine learning , volume 1
Tom Mitchell. 1997 · 1997
Earlier work this paper cites.
A dataset for statutory reasoning in tax law entailment and question answering
Original
Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020 · 2005
Earlier work this paper cites.
Neural legal judgment prediction in english
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019 · 2019
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
Billsum: A corpus for automatic summarization of us legislation
Anastassia Kornilova and Vladimir Eidelman. 2019 · 2019
Earlier work this paper cites.
Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019 · 2019
Earlier work this paper cites.
Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019 · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020 · 2020
Earlier work this paper cites.

Similar

Legal judgement prediction for uk courts
Benjamin Strickson and Beatriz De La Iglesia. 2020 · 2020
Cited alongside, same era.
Iteratively questioning and answering for interpretable legal judgment prediction
Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020 · 2020
Cited alongside, same era.
Multi-task legal judgement prediction combining a subtask of seriousness of charge
Xu Zhuopeng, Li Xia, Li Yinlin, Wang Zihan, Fanxu Yujie, and Lai Xiaoyan. 2020 · 2020
Cited alongside, same era.
Justice: A benchmark dataset for supreme court’s judgment prediction
Original
Mohammad Alali, Shaayan Syed, Mohammed Alsayed, Smit Patel, and Hemanth Bodala. 2021 · 2021
Cited alongside, same era.
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021 · 2021
Cited alongside, same era.
Cuad: An expert-annotated nlp dataset for legal contract review
Original
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021 · 2021
Cited alongside, same era.
Ildc for cjpe: Indian legal documents corpus for court judgment prediction and explanation
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021 · 2021
Cited alongside, same era.
Swiss-judgment-prediction: A multilingual legal judgment prediction benchmark
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021 · 2021
Cited alongside, same era.
Multitask prompted training enables zero-shot task generalization
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021 · 2021
Cited alongside, same era.
Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning
Original
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022a
Cited in the paper.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
Original
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a
Cited in the paper.
P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022b
Cited in the paper.

Then

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Ben Wang and Aran Komatsuzaki. 2021 · 2021
Later among the works it cites.
Promptsource: An integrated development environment and repository for natural language prompts
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. 2022 · 2022
Closest in time.
Gpt-neox-20b: An open-source autoregressive language model
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022 · 2022
Closest in time.
Revisiting transformer-based models for long document classification
Original
Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. 2022 · 2022
Closest in time.
Openprompt: An open-source framework for prompt-learning
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022 · 2022
Closest in time.
Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022 · 2022
Closest in time.
mgpt: Few-shot learners go multilingual
Original
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022 · 2022
Closest in time.
Multi-label legal document classification: A deep learning-based approach with label-attention and domain-specific pre-training
Dezhao Song, Andrew Vold, Kanika Madan, and Frank Schilder. 2022 · 2022
Closest in time.
On the role of negative precedent in legal outcome prediction
Original
Josef Valvoda, Ryan Cotterell, and Simone Teufel. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…