Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

Understand

Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence, and leveraging LLMs to identify inconsistencies in law.

• This paper explores LLM capabilities in applying tax law.
• We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies.
• Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release.