2022

Case law retrieval: problems, methods, challenges and evaluations in the last 20 years

Locke, Daniel, Zuccon, Guido

Understand

Case law retrieval is the retrieval of judicial decisions relevant to a legal question.

Case law retrieval comprises a significant amount of a lawyer's time, and is important to ensure accurate advice and reduce workload.
We survey methods for case law retrieval from the past 20 years and outline the problems and challenges facing evaluation of case law retrieval systems going forward.
Limited published work has focused on improving ranking in ad-hoc case law retrieval.

Built on

Computer Retrieval of Case Law
Robert A Wilson. 1962 · 1962
Earlier work this paper cites.
Bibliographic Coupling Between Scientific Papers
Maxwell Mirton Kessler. 1963 · 1963
Earlier work this paper cites.
Legal Information Retrieval by Computer: Applications and Implications
Colin Tapper. 1974 · 1974
Earlier work this paper cites.
The Use of Citation Vectors for Legal Information Retrieval
Colin Tapper. 1981 · 1981
Earlier work this paper cites.
ASK for Information Retrieval: Part I. Background and Theory
Nicholas J Belkin, Robert N Oddy, and Helen M Brooks. 1982 · 1982
Earlier work this paper cites.
A Brief History of Computer-Assisted Legal Research
William G Harrington. 1984 · 1984
Earlier work this paper cites.
An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System
David C Blair and Melvin E Maron. 1985 · 1985
Earlier work this paper cites.
The Curse of Thamus: An Analysis of Full-text Legal Document Retrieval
Daniel P Dabney. 1986 · 1986
Earlier work this paper cites.
Australasian Computerised Legal Information Handbook
Graham Greenleaf, Andrew Mowbary, and David Lewis. 1988 · 1988
Earlier work this paper cites.
Toward a Comprehensive Legal Information Retrieval System. In Database and Expert Systems Applications
Daphne Gelbart and JC Smith. 1990 · 1990
Earlier work this paper cites.
Beyond Boolean Search: FLEXICON, a Legal Tex-based Intelligent System. In Proceedings of the 3rd International Conference on Artificial intelligence and Law
Dephne Gelbart and JC Smith. 1991 · 1991
Earlier work this paper cites.
Inference Networks for Document Retrieval
Howard Turtle. 1991 · 1991
Earlier work this paper cites.
Evaluation of an Inference Network-Based Retrieval Model
Howard Turtle and W Bruce Croft. 1991 · 1991
Earlier work this paper cites.
The INQUERY Retrieval System. In Database and Expert Systems Applications
James P Callan, W Bruce Croft, and Stephen M Harding. 1992 · 1992
Earlier work this paper cites.
Statistical Modeling of Relevance Judgments for Probabilistic Retrieval of American Case Law
Daniel P Dabney. 1993 · 1993
Earlier work this paper cites.
Natural Language vs. Boolean Query Evaluation: A Comparison of Retrieval Performance. In Proceedings of the 17th International ACM SIGIR Conference on Research and Development in Information Retrieval
Howard Turtle. 1994 · 1994
Earlier work this paper cites.
An Intelligent Interface for Legal Databases. In Proceedings of the 5th International Conference on Artificial Intelligence and Law
Luuk Matthijssen. 1995 · 1995
Earlier work this paper cites.
Text Retrieval in the Legal World
Howard Turtle. 1995 · 1995
Earlier work this paper cites.
The Synergistic Application of CBR to IR
Edwina L Rissland and Jody J Daniels. 1996 · 1996
Earlier work this paper cites.
A Task-Based Interface to Legal Databases
Luuk Matthijssen. 1998 · 1998
Earlier work this paper cites.
Information Retrieval as Statistical Translation. In Proceedings of the 22nd International ACM SIGIR Conference on Research and Development in Information Retrieval
Adam Berger and John Lafferty. 1999 · 1999
Earlier work this paper cites.
Evaluating Legal Information Retrieval Systems: How do the Ranked-retrieval Methods of WESTLAW and LEXIS Measure Up?
Kevin Gerson. 1999 · 1999
Earlier work this paper cites.
The PageRank Citation Ranking: Bringing Order to the Web
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999 · 1999
Earlier work this paper cites.
Innovative Techniques for Legal Text Retrieval
Marie-Francine Moens. 2001 · 2001
Earlier work this paper cites.
Ontologies for Legal Information Serving and Knowledge Management. In Proceedings of the 15th Annual Conference on Legal Knowledge and Information Systems
Joost Breuker, Abdullatif Elhag, Emil Petkov, and Radboud Winkels. 2002 · 2002
Earlier work this paper cites.
Information Extraction from Case Law and Retrieval of Prior Cases
Peter Jackson, Khalid Al-Kofahi, Alex Tyrrell, and Arun Vachher. 2003 · 2003
Earlier work this paper cites.
Legal Ontologies in Knowledge Engineering and Information Management
Joost Breuker, André Valente, and Radboud Winkels. 2004 · 2004
Earlier work this paper cites.
The Legal Concepts and the Layman’s Terms - Bridging the Gap through Ontology-Based Reasoning about Liability. In Proceedings of the 18th Conference on Legal Knowledge and Information Systems
Ronny van Laarschot, Wouter Van Steenbergen, Heiner Stuckenschmidt, Arno R. Lodder, and Frank van Harmelen. 2005 · 2005
Earlier work this paper cites.
TREC: Experiment and Evaluation in Information Retrieval
Ellen M Voorhees, Donna K Harman, et al · 2005
Earlier work this paper cites.
A Cross-language Information Retrieval Based on an Arabic Ontology in the Legal Domain. In Proceedings of the International Conference on Signal-Image Technology and Internet-Based Systems
Soraya Zaidi, Mohamed Tayeb Laskri, and K Bechkoum. 2005 · 2005
Earlier work this paper cites.
TREC 2006 Legal Track Overview. In Proceedings of the 15th Text REtrieval Conference
Jason R Baron, David D Lewis, and Douglas W Oard. 2006 · 2006
Earlier work this paper cites.
Extractive Summarisation of Legal Texts
Ben Hachey and Claire Grover. 2006 · 2006
Earlier work this paper cites.
Thesaurus-based Retrieval of Case Law. In Proceedings of the 2006 Conference on Legal Knowledge and Information Systems
Michel CA Klein, Wouter van Steenbergen, Elisabeth M Uijttenbroek, Arno R Lodder, and Frank van Harmelen. 2006 · 2006
Earlier work this paper cites.
Legal Information Retrieval Study: Lexis Professional and Westlaw UK
Dean Mason. 2006 · 2006
Earlier work this paper cites.
Improving Legal Document Summarization using Graphical Models
M Saravanan, Balaraman Ravindran, and S Raman. 2006 · 2006
Earlier work this paper cites.
In ICAIL 2007 DESI Workshop: Supporting Search and Sensemaking for Electronically Stored Information in Discovery Proceedings
Jason R Baron, Douglas W Oard, David D Lewis, and Paul Thompson. 2007 · 2007
Earlier work this paper cites.
A new approach for evaluating query expansion: query-document term mismatch. In Proceedings of the 30th International ACM SIGIR Conference on Research and Development in Information Retrieval
Tonya Custis and Khalid Al-Kofahi. 2007 · 2007
Earlier work this paper cites.
Network Analysis and the Law: Measuring the Legal Importance of Precedents at the US Supreme Court
James H Fowler, Timothy R Johnson, James F Spriggs, Sangick Jeon, and Paul J Wahlbeck. 2007 · 2007
Earlier work this paper cites.
Similarity Measures for Short Segments of Text. In European Conference on Information Retrieval
Donald Metzler, Susan Dumais, and Christopher Meek. 2007 · 2007
Earlier work this paper cites.
Legal Query Expansion Using Ontologies and Relevance Feedback. In Proceedings of the Workshop on Legal Ontologies and Artificial Intelligence Techniques
Erich Schweighofer and Anton Geist. 2007 · 2007
Earlier work this paper cites.
Semantics-based Legal Citation Network. In Proceedings of the 11th International Conference on Artificial Intelligence and Law
Paul Zhang and Lavanya Koppaka. 2007 · 2007
Earlier work this paper cites.
Investigating External Corpus and Clickthrough Statistics for Query Expansion in the Legal Domain. In Proceedings of the 17th ACM Conference on Information and Knowledge Management
Tonya Custis and Khalid Al-Kofahi. 2008 · 2008
Earlier work this paper cites.
Concept and Context in Legal Information Retrieval. In Proceedings of the 21st Conference on Legal Knowledge and Information Systems
Tamsin Maxwell and Burkhard Schafer. 2008 · 2008
Earlier work this paper cites.
Retrieval of Case Law to Provide Layman with Information about Liability: Preliminary Results of the BEST-Project
Elisabeth Uijttenbroek, Arno Lodder, Michel Klein, Gwen Wildeboer, Wouter Van Steenbergen, Rory Sie, Paul Huygen, and Frank van Harmelen. 2008 · 2008
Earlier work this paper cites.
Using Citation Analysis Techniques for Computer-Assisted Legal Research in Continental Jurisdictions
Anton Geist. 2009 · 2009
Earlier work this paper cites.

Similar

Feature Engineering on Event-Centric Surrogate Documents to Improve Search Results. In Proceedings of the 18th ACM Conference on Information and Knowledge Management
Wenhui Liao and Isabelle Moulinier. 2009 · 2009
Cited alongside, same era.
Improving Legal Information Retrieval using an Ontological Framework
M. Saravanan, B. Ravindran, and S. Raman. 2009 · 2009
Cited alongside, same era.
Query Expansion Algorithm for Supreme Court Sentences Retrieval Using Ontology. In Proceedings of the 48th Kasetsart Annual Conference
Tanapon Tantisripreecha and Nuanwan Soonthornphisaj. 2009 · 2009
Cited alongside, same era.
Let there be LITE: a Brief History of Legal Information Retrieval
Jon Bing. 2010 · 2010
Cited alongside, same era.
Thai Succession and Family Law Ontology Building Using Ant Colony Algorithm. In New Frontiers in Artificial Intelligence
Vi-sit Boonchom and Nuanwan Soonthornphisaj. 2010 · 2010
Cited alongside, same era.
Clinical Information Retrieval Using Document and PICO Structure. In Human Language Technologies: The 2010 Conference of the North American Chapter of the Association for Computational Linguistics
Florian Boudin, Jian-Yun Nie, and Martin Dawes. 2010 · 2010
Cited alongside, same era.
Overview of the TREC 2010 Legal Track. In Proceedings of the 19th Text REtrieval Conference
Gordon V Cormack, Maura R Grossman, Bruce Hedin, and Douglas W Oard. 2010 · 2010
Cited alongside, same era.
Technology-Assisted Review in e-Discovery can be More Effective and More Efficient than Exhaustive Manual Review
Maura R Grossman and Gordon V Cormack. 2010 · 2010
Cited alongside, same era.
Estimation of Statistical Translation Models Based on Mutual Information for Ad Hoc Information Retrieval. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval
Maryam Karimzadehgan and ChengXiang Zhai. 2010 · 2010
Cited alongside, same era.
Overview of ResPubliQA 2009: Question Answering Evaluation Over European Legislation
Anselmo Peñas, Pamela Forner, Richard Sutcliffe, Álvaro Rodrigo, Corina Forăscu, Iñaki Alegria, Danilo Giampiccolo, Nicolas Moreau, and Petya Osenova. 2010 · 2010
Cited alongside, same era.
Test Collection Based Evaluation of Information Retrieval Systems
Mark Sanderson. 2010 · 2010
Cited alongside, same era.
Query Expansion for the Legal Domain: A Case Study from the JUMAS Project
Fabio Sartori and Matteo Palmonari. 2010 · 2010
Cited alongside, same era.

Then

Analyzing the Extraction of Relevant Legal Judgments using Paragraph-level and Citation Information
K Raghav, P Krishna Reddy, and V Balakista Reddy. 2016 · 2016
Later among the works it cites.
Recognizing Cited Facts and Principles in Legal Judgements
Olga Shulayeva, Advaith Siddharthan, and Adam Wyner. 2016 · 2016
Later among the works it cites.
Legal Text Retrieval–Semantic Web Approach
Jayasudha Subburaj, C Soundarya Veni, and R Nandhini Amirtha. 2016 · 2016
Later among the works it cites.
On the Concept of Relevance in Legal Information Retrieval
Marc van Opijnen and Cristiana Santos. 2016 · 2016
Later among the works it cites.
A Scalable Approach to Legal Question Answering. In Proceedings of the 16th International Conference on Articial Intelligence and Law
Zachary Bennett, Tony Russell-Rose, and Kate Farmer. 2017 · 2017
Later among the works it cites.
Lexical-Morphological Modeling for Legal Text Analysis
Danilo S. Carvalho, Minh-Tien Nguyen, Chien-Xuan Tran, and Minh-Le Nguyen. 2017 · 2017
Later among the works it cites.
Scenario Analytics: Analyzing Jury Verdicts to Evaluate Legal Case Outcomes. In Proceedings of the 16th International Conference on Artificial Intelligence and Law
Jack G Conrad and Khalid Al-Kofahi. 2017 · 2017
Later among the works it cites.
LexrideLaw: an Argument Based Legal Search Engine. In Proceedings of the 16th International Conference on Artificial Intelligence and Law
Matthew Gifford. 2017 · 2017
Later among the works it cites.
Legal Content Fusion for Legal Information Retrieval. In Proceedings of the 16th International Conference on Artificial Intelligence and Law
Seongwan Heo, Kihyun Hong, and Young-Yik Rhim. 2017 · 2017
Later among the works it cites.
Two-step Cascaded Textual Entailment for Legal Bar Exam Question Answering. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law
Mi-Young Kim and Randy Goebel. 2017 · 2017
Later among the works it cites.
Applying a Convolutional Neural Network to Legal Question Answering
Mi-Young Kim, Ying Xu, and Randy Goebel. 2017 · 2017
Later among the works it cites.
Evaluation of Diversification Techniques for Legal Information Retrieval
Marios Koniaris, Ioannis Anagnostopoulos, and Yannis Vassiliou. 2017 · 2017
Later among the works it cites.
Generating clinical queries from patient narratives. In Proceedings of the 40th international ACM SIGIR conference on Research and Development in Information Retrieval
Bevan Koopman, Liam Cripwell, and Guido Zuccon. 2017 · 2017
Later among the works it cites.
Detection of Catchphrases and Precedence in Legal Documents. In Working notes of FIRE 2017-Forum for Information Retrieval Evaluation
Yogesh Kulkarni, Rishabh Patil, and Srinivasan Shridharan. 2017 · 2017
Later among the works it cites.
Automatic Cited Decision Retrieval: Working Notes of Ielab for FIRE Legal Track Precedence Retrieval Task. In Working notes of FIRE 2017-Forum for Information Retrieval Evaluation (CEUR Workshop Proceedings)
Daniel Locke and Guido Zuccon. 2017 · 2017
Later among the works it cites.
Automatic Query Generation from Legal Texts for Case Law Retrieval
Daniel Locke, Guido Zuccon, and Harrisen Scells. 2017 · 2017
Later among the works it cites.
A Semi-Supervised Training Method for Semantic Search of Legal Facts in Canadian Immigration Cases
Isar Nejadgholi, Renaud Bougueng, and Samuel Witherspoon. 2017 · 2017
Later among the works it cites.
Giving Every Case its (Legal) Due: The Contribution of Citation Networks and Text Similarity Techniques to Legal Studies of European Union law. In Proceedings of the 30th International Conference on Legal Knowledge and Information Systems
Yannis Panagis, Urska Sadl, and Fabien Tarissan. 2017 · 2017
Later among the works it cites.
Integrating the Framing of Clinical Questions via PICO into the Retrieval of Medical Literature for Systematic Reviews. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management
Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, and Shlomo Geva. 2017 · 2017
Later among the works it cites.
HLJIT2017@IRLed-FIRE2017: Information Retrieval From Legal Documents. In Working notes of FIRE 2017-Forum for Information Retrieval EvaluationRab
Luiyang Tian, Hui Ning, Leilei Kong, Zongyuan Han, Ruiming Xiao, and Haoliang Qi. 2017 · 2017
Later among the works it cites.
Experimenting Word Embeddings in Assisting Legal Review. In Proceedings of the 16th International Conference on Articial Intelligence and Law
Ngoc Phuoc An Vo, Caroline Privault, and Fabien Guillot. 2017 · 2017
Later among the works it cites.
Ravel Judge Analytics
2018 · 2018
Later among the works it cites.
COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment. In New Frontiers in Artificial Intelligence
Yoshinobu Kano, Mi-Young Kim, Masaharu Yoshioka, Yao Lu, Juliano Rabelo, Naoki Kiyota, Randy Goebel, and Ken Satoh. 2019 · 2018
Later among the works it cites.
Network Analysis in the Legal Domain: A Complex Model for European Union Legal Sources
Marios Koniaris, Ioannis Anagnostopoulos, and Yannis Vassiliou. 2018 · 2018
Later among the works it cites.
A Test Collection for Evaluating Legal Case Law Search. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval
Daniel Locke and Guido Zuccon. 2018 · 2018
Later among the works it cites.
Information Retrieval in the Workplace: A Comparison of Professional Search Practices
Tony Russell-Rose, Jon Chamberlain, and Leif Azzopardi. 2018 · 2018
Later among the works it cites.
Deep Ensemble Learning for Legal Query Understanding. In International Workshop on Legal Data Analytics and Mining
Arunprasath Shankar and Venkata Nagaraju Buddarapu. 2018 · 2018
Later among the works it cites.
Modeling Dynamic Pairwise Attention for Crime Classification over Legal Articles. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
Pengfei Wang, Ze Yang, Shuzi Niu, Yongfeng Zhang, Lei Zhang, and ShaoZhang Niu. 2018 · 2018
Later among the works it cites.
Senior Software Engineer - Search, Recommendation and Data infrastructure
2019 · 2019
Later among the works it cites.
Overview of the FIRE 2019 AILA Track: Artificial Intelligence for Legal Assistance.. In FIRE (Working Notes)
Paheli Bhattacharya, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, Parth Mehta, Arnab Bhattacharya, and Prasenjit Majumder. 2019 · 2019
Later among the works it cites.
Data Shift in Legal AI Systems.. In Proceedings of the 17th International Conference on Artificial Intelligence and Law
Venkata Nagaraju Buddarapu and Arunprasath Shankar. 2019 · 2019
Later among the works it cites.
Westlaw Edge AI Features Demo: KeyCite Overruling Risk, Litigation Analytics, and WestSearch Plus. In Proceedings of the 17th International Conference on Artificial Intelligence and Law
Tonya Custis, Frank Schilder, Thomas Vacek, Gayle McElvain, and Hector Martinez Alonso. 2019 · 2019
Later among the works it cites.
Inside the Black Box of Search Algorithms
Susan Nevelow Mart, Joe Breda, Ed Walters, Tito Sierra, and Khalid Al-Kofahi. 2019 · 2019
Later among the works it cites.
WestSearch Plus: A Non-Factoid Question-Answering System for the Legal Domain. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
Gayle McElvain, George Sanchez, Sean Matthews, Don Teo, Filippo Pompili, and Tonya Custis. 2019 · 2019
Later among the works it cites.
Exploiting Search Logs to Aid in Training and Automating Infrastructure for Question Answering in Professional Domains. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law
Filippo Pompili, Jack G Conrad, and Carter Kolbeck. 2019 · 2019
Later among the works it cites.
A Summary of the COLIEE 2019 Competition. In JSAI International Symposium on Artificial Intelligence
Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. 2019 · 2019
Later among the works it cites.
Searching Case Law Judgements by Using Other Judgements as a Query
Sami SARSA and Eero Hyvonen. 2019 · 2019
Later among the works it cites.
Improving Sentence Retrieval from Case Law for Statutory Interpretation. In Proceedings of the 17th International Conference on Artificial Intelligence and Law
Jaromir Savelka, Huihui Xu, and Kevin D Ashley. 2019 · 2019
Later among the works it cites.
Patent retrieval: a literature review
Walid Shalaby and Wlodek Zadrozny. 2019 · 2019
Later among the works it cites.
Legal Query Reformulation using Deep Learning. In Proceedings of the 3rd Workshop on Automated Semantic Analysis of Information in Legal Text
Arunprasath Shankar and Venkata Nagaraju Buddarapu. 2019 · 2019
Later among the works it cites.
Hierarchical Matching Network for Crime Classification. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
Pengfei Wang, Yu Fan, Shuzi Niu, Ze Yang, Yongfeng Zhang, and Jiafeng Guo. 2019 · 2019
Later among the works it cites.
Citation Metrics for Legal Information Retrieval Systems
Gineke Wiggers and Suzan Verberne. 2019 · 2019
Later among the works it cites.
Open Domain Web Keyphrase Extraction Beyond Language Modeling
Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, and Arnold Overwijk. 2019 · 2019
Later among the works it cites.
Hier-SPCNet: A Legal Statute Hierarchy-based Heterogeneous Network for Computing Legal Case Document Similarity. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
Paheli Bhattacharya, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh. 2020 · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…