2020

A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

Holzenberger, Nils, Blair-Stanek, Andrew, Van Durme, Benjamin

Understand

Legislation can be viewed as a body of prescriptive rules expressed in natural language.

The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language.
Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics.
To investigate the performance of natural language understanding approaches on statutory reasoning, we introduce a dataset, together with a legal-domain text corpus.

Built on

Reasoning foundations of medical diagnosis
Robert S Ledley and Lee B Lusted. 1959 · 1959
Earlier work this paper cites.
Comment on D. Davidson’s “The logical forms of action sentences”
Hector Neri Castañeda. 1967 · 1967
Earlier work this paper cites.
The logical forms of action sentences
Donald Davidson. 1967 · 1967
Earlier work this paper cites.
Judith, a computer program to advise lawyers in reasoning a case
Walter G Popp and Bernhard Schlink. 1974 · 1974
Earlier work this paper cites.
A model of inexact reasoning in medicine
Edward H Shortliffe and Bruce G Buchanan. 1975 · 1975
Earlier work this paper cites.
Reflections on TAXMAN: An experiment in artificial intelligence and legal reasoning
L Thorne McCarty. 1976 · 1976
Earlier work this paper cites.
A computer program for legal planning and analysis: Taxation of stock redemptions
Robert Hellawell. 1980 · 1980
Earlier work this paper cites.
Internist-I, an experimental computer-based diagnostic consultant for general internal medicine
Randolph A Miller, Harry E Pople Jr, and Jack D Myers. 1982 · 1982
Earlier work this paper cites.
The design of a legal analysis program. In AAAI-83 . 114–118
Anne vdL Gardner. 1983 · 1983
Earlier work this paper cites.
The British Nationality Act as a logic program
Marek J. Sergot, Fariba Sadri, Robert A. Kowalski, Frank Kriwaczek, Peter Hammond, and H Terese Cory. 1986 · 1986
Earlier work this paper cites.
Logic programming for large scale applications in law: A formalisation of supplementary benefit legislation. In Proceedings of the 1st international conference on Artificial intelligence and law . 190–198
Trevor JM Bench-Capon, Gwen O Robinson, Tom W Routen, and Marek J Sergot. 1987 · 1987
Earlier work this paper cites.
A Prolog model of the income tax act of Canada. In Proceedings of the 1st international conference on Artificial intelligence and law . 127–136
David M Sherman. 1987 · 1987
Earlier work this paper cites.
Events in the Semantics of English . Vol. 334
Terence Parsons. 1990 · 1990
Earlier work this paper cites.
Expert systems: principles and practice
Edward A Feigenbaum. 1992 · 1992
Earlier work this paper cites.
Using the framework
Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al · 1996
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001 · 2001
Earlier work this paper cites.
Predicting outcomes of case based legal arguments. In Proceedings of the 9th international conference on Artificial intelligence and law . 233–242
Stefanie Bruninghaus and Kevin D Ashley. 2003 · 2003
Earlier work this paper cites.
AI and Law: A fruitful synergy
Edwina L Rissland, Kevin D Ashley, and Ronald Prescott Loui. 2003 · 2003
Earlier work this paper cites.
Project halo: Towards a digital aristotle
Noah S Friedland, Paul G Allen, Gavin Matthews, Michael Witbrock, David Baxter, Jon Curtis, Blake Shepard, Pierluigi Miraglia, Jurgen Angele, Steffen Staab, et al · 2004
Earlier work this paper cites.
Automatic semantics extraction in law documents. In Proceedings of the 10th international conference on Artificial intelligence and law . 133–140
Carlo Biagioli, Enrico Francesconi, Andrea Passerini, Simonetta Montemagni, and Claudia Soria. 2005 · 2005
Earlier work this paper cites.
The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop . Springer, 177–190
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005 · 2005
Earlier work this paper cites.
The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006 · 2006
Earlier work this paper cites.

Similar

The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing . Association for Computational Linguistics, 1–9
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007 · 2007
Cited alongside, same era.
Open information extraction from the web
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008 · 2008
Cited alongside, same era.
Automatically classifying case texts and predicting outcomes
Kevin D Ashley and Stefanie Brüninghaus. 2009 · 2009
Cited alongside, same era.
The Fifth PASCAL Recognizing Textual Entailment Challenge.. In TAC
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009 · 2009
Cited alongside, same era.
Building Watson: An overview of the DeepQA project
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al · 2010
Cited alongside, same era.
Project Halo Update—Progress Toward Digital Aristotle
David Gunning, Vinay K Chaudhri, Peter E Clark, Ken Barker, Shaw-Yi Chaw, Mark Greaves, Benjamin Grosof, Alice Leung, David D McDonald, Sunil Mishra, et al · 2010
Cited alongside, same era.
PROLEG: an implementation of the presupposed ultimate fact theory of Japanese civil code by PROLOG technology. In JSAI International Symposium on Artificial Intelligence . Springer, 153–164
Ken Satoh, Kento Asai, Takamune Kogawa, Masahiro Kubota, Megumi Nakamura, Yoshiaki Nishigai, Kei Shirakawa, and Chiaki Takano. 2010 · 2010
Cited alongside, same era.
Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011 · 2011
Cited alongside, same era.
Learning to win by reading manuals in a monte-carlo framework
SRK Branavan, David Silver, and Regina Barzilay. 2012 · 2012
Cited alongside, same era.
The concept of law
Herbert Lionel Adolphus Hart and Herbert Lionel Adolphus Hart. 2012 · 2012
Cited alongside, same era.
Automatically Extracting Procedural Knowledge from Instructional Texts using Natural Language Processing.. In LREC , Vol. 2012. 520–527
Ziqi Zhang, Philip Webster, Victoria S Uren, Andrea Varga, and Fabio Ciravegna. 2012 · 2012
Cited alongside, same era.
Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems . 3111–3119
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013 · 2013
Cited alongside, same era.

Then

Never-ending learning
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhanava Dalvi, Matt Gardner, Bryan Kisiel, et al · 2018
Later among the works it cites.
Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 · 2018
Later among the works it cites.
Interpretation of natural language rules in conversational machine reading
Original
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018 · 2018
Later among the works it cites.
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Brussels, Belgium, 2369–2380
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018 · 2018
Later among the works it cites.
Overview of japanese statute law retrieval and entailment task at coliee-2018. In Twelfth International Workshop on Juris-informatics (JURISIN 2018)
Masaharu Yoshioka, Yoshinobu Kano, Naoki Kiyota, and Ken Satoh. 2018 · 2018
Later among the works it cites.
Swag: A large-scale adversarial dataset for grounded commonsense inference
Original
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018 · 2018
Later among the works it cites.
Caselaw Access Project
2019 · 2019
Later among the works it cites.
Large-Scale Multi-Label Text Classification on EU Legislation
Original
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2019 · 2019
Later among the works it cites.
From’F’to’A’on the NY Regents Science Exams: An Overview of the Aristo Project
Original
Peter Clark, Oren Etzioni, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, Sumithra Bhakthavatsalam, et al · 2019
Later among the works it cites.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs
Original
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019 · 2019
Later among the works it cites.
Black’s Law Dictionary (11 ed.)
Bryan A Gardner. 2019 · 2019
Later among the works it cites.
Statute Law Information Retrieval and Entailment. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law . 283–289
Mi-Young Kim, Juliano Rabelo, and Randy Goebel. 2019 · 2019
Later among the works it cites.
BillSum: A Corpus for Automatic Summarization of US Legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization . Association for Computational Linguistics, Hong Kong, China, 48–56
Anastassia Kornilova and Vladimir Eidelman. 2019 · 2019
Later among the works it cites.
Combining Similarity and Transformer Methods for Case Law Entailment. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law . 290–296
Juliano Rabelo, Mi-Young Kim, and Randy Goebel. 2019 · 2019
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 2019
Later among the works it cites.
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Original
Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019 · 2019
Later among the works it cites.
Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments. Association for Computational Linguistics, Minneapolis, Minnesota
Jerrold Soh, How Khang Lim, and Ian Ernst Chai. 2019 · 2019
Later among the works it cites.
Mining User-Generated Repair Instructions from Automotive Web Communities. In Proceedings of the 52nd Hawaii International Conference on System Sciences
Thiemo Wambsganß and Hansjörg Fromm. 2019 · 2019
Later among the works it cites.
Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems . 3261–3275
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019 · 2019
Later among the works it cites.
Broad-Coverage Semantic Parsing as Transduction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Association for Computational Linguistics, Hong Kong, China, 3786–3798
Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019 · 2019
Later among the works it cites.
JEC-QA: A Legal-Domain Question Answering Dataset
Original
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2019 · 2019
Later among the works it cites.
Towards Evaluating Plan Generation Approaches with Instructional Texts
Original
Debajyoti Paul Chowdhury, Arghya Biswas, Tomasz Sosnowski, and Kristina Yordanova. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…