2023

Auditing large language models: a three-layered approach

Mökander, Jakob, Schuett, Jonas, Kirk, Hannah Rose et al.

Understand

Large language models (LLMs) represent a major advance in artificial intelligence (AI) research.

However, the widespread use of LLMs is also coupled with significant ethical and social challenges.
Previous research has pointed towards auditing as a promising governance mechanism to help ensure that AI systems are designed and deployed in ways that are ethical, legal, and technically robust.
However, existing auditing procedures fail to address the governance challenges posed by LLMs, which display emergent capabilities and are adaptable to a wide range of downstream tasks.

Built on

“Representing and Intervening : Introductory Topics in the Philosophy of Natural Science”
Ian Hacking · 1983
Earlier work this paper cites.
“The Pursuit of the Ideal”
Isaiah Berlin · 1988
Earlier work this paper cites.
“Philosophy and principles of auditing : an introduction”
David Flint · 1988
Earlier work this paper cites.
“Natural Language Processing”
Aravind. Joshi · 1991
Earlier work this paper cites.
“The Audit Society : Rituals of Verification”
Michael Power · 1997
Earlier work this paper cites.
“Applied Policy Research : Concepts and Cases”
Peter Haas and J. Springer · 1998
Earlier work this paper cites.
“Two Decades of Statistical Language Modeling Where Do We Go Form Here? Where Do We Go from Here?”
Ronald Rosenfeld · 2000
Earlier work this paper cites.
“Towards a Human-like Open-Domain Chatbot”
Daniel Adiwardana et al · 2001
Earlier work this paper cites.
“Scaling Laws for Neural Language Models”
Jared Kaplan et al · 2001
Earlier work this paper cites.
“Risk Management Guide for Information Technology Systems Recommendations of the National Institute of Standards and Technology”
NIST · 2002
Earlier work this paper cites.
“Gender Bias in Coreference Resolution”
Rachel Rudinger, Jason Naradowsky, Brian Leonard and Benjamin Van · 2002
Earlier work this paper cites.
“Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims”
Miles Brundage et al · 2004
Earlier work this paper cites.
“Comtemporary Auditing”
Kamal Gupta · 2004
Earlier work this paper cites.
“Language Models Are Few-Shot Learners”
Tom. Brown et al · 2005
Earlier work this paper cites.
“Differential privacy”
Cynthia Dwork · 2006
Earlier work this paper cites.
“The evolution of auditing: An analysis of the historical development”
Teck-Heang Lee and Azham Md · 2008
Earlier work this paper cites.
“Auditing Algorithms”
Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort · 2009
Earlier work this paper cites.
“Learning to summarize from human feedback”
Original
Nisan Stiennon et al · 2009
Earlier work this paper cites.
“Information Technology Control and Audit”
Sandra Senft and Frederick Gallegos · 2009
Earlier work this paper cites.
“The Methods of Business Ethics”
Ronald. Green and Aine Donovan · 2009
Earlier work this paper cites.
“A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies”
Maria. Grant and Andrew Booth · 2009
Earlier work this paper cites.
“The Re-Emergence of Emergence, and the Causal Role of Synergy in Emergent Evolution”
Peter. Corning · 2010
Earlier work this paper cites.
“Auditing IT Governance”
Florin-Mihai Iliescu · 2010
Earlier work this paper cites.
“Measuring and Reducing Gendered Correlations in Pre-trained Models”
Kellie Webster et al · 2010
Earlier work this paper cites.
“Encyclopedia of Research Design”
Neil Salkind · 2010
Earlier work this paper cites.
“Understanding Regulation: Theory, Strategy, and Practice”
Robert Baldwin, Martin Cave and Martin Lodge · 2011
Earlier work this paper cites.
“Engineering a Safer World : Systems Thinking Applied to Safety”
Nancy Leveson · 2011
Earlier work this paper cites.
“Legal Compliance”
Samuel. Idowu · 2013
Earlier work this paper cites.
“Truth-Telling by Third-Party Auditors and the Response of Polluting Firms: Experimental Evidence from India*”
Esther Duflo, Michael Greenstone, Rohini Pande and Nicholas Ryan · 2013
Earlier work this paper cites.
“Algorithmic Accountability: Journalistic Investigation of Computational Power Structures”
Nicholas Diakopoulos · 2014
Earlier work this paper cites.
“The Generative Internet”
Jonathan. Zittrain · 2014
Earlier work this paper cites.
“Research Design”
Eliot Smith · 2014
Earlier work this paper cites.
“Reputation and Power: Organizational Image and Pharmaceutical Regulation at the FDA”
Daniel Carpenter · 2014
Earlier work this paper cites.
“Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering”
Claes Wohlin · 2014
Earlier work this paper cites.
“ImageNet Large Scale Visual Recognition Challenge”
Olga Russakovsky et al · 2015
Earlier work this paper cites.
“Advances in Natural Language Processing”
Julia Hirschberg and Christopher. Manning · 2015
Earlier work this paper cites.
“Artificial Intelligence : A Modern Approach”, 2015
Stuart Russell and Peter Norvig · 2015
Earlier work this paper cites.
“Ethics in Business Practice and Regulation”
Christopher Hodges · 2015
Earlier work this paper cites.
“ISO/IEC 38500:2015 - Information Technology — Governance of IT for the Organization”
ISO · 2015
Earlier work this paper cites.
“Quality Management Audits in Nuclear Medicine Practices”
IAEA · 2015
Earlier work this paper cites.
“Conducting Semi-Structured Interviews”
William. Adams · 2015
Earlier work this paper cites.
“Generative Models”, 2016
OpenAI · 2016
Earlier work this paper cites.
“Software Certification Services: Encouraging Trust and Reasonable Expectations”
Jeffrey Voas and Keith Miller · 2016
Earlier work this paper cites.
“Auditing for Transparency in Content Personalization Systems”
Brent Mittelstadt · 2016
Earlier work this paper cites.
“The Necessity of Auditing Artificial Intelligence”
Jennifer Bauer · 2016
Earlier work this paper cites.
“AI Assisted Ethics”
Amitai Etzioni and Oren Etzioni · 2016
Earlier work this paper cites.
“Infraethics–on the Conditions of Possibility of Morality”
Luciano Floridi · 2017
Earlier work this paper cites.
“Attention is All You Need”
Ashish Vaswani et al · 2017
Earlier work this paper cites.
“Counterfactual Fairness”
Matt Kusner, Joshua Loftus, Chris Russell and Ricardo Silva · 2017
Earlier work this paper cites.
“Regulating Autonomous Systems: Beyond Standards”
David Danks and Alex London · 2017
Earlier work this paper cites.
“Responsible Autonomy”
Virginia Dignum · 2017
Earlier work this paper cites.
“Deep Reinforcement Learning from Human Preferences”
Original
Paul Christiano et al · 2017
Earlier work this paper cites.
“Challenges for Transparency”
Adrian Weller · 2017
Earlier work this paper cites.
“Semantics Derived Automatically from Language Corpora Contain Human-like Biases”
Aylin Caliskan, Joanna. Bryson and Arvind Narayanan · 2017
Earlier work this paper cites.
“AI Governance: A Research Agenda”, 2017, pp. 1–53
Allan Dafoe · 2017
Earlier work this paper cites.
“Detecting Online Hate Speech Using Context Aware Models”
Lei Gao and Ruihong Huang · 2017
Earlier work this paper cites.
“The Ethics of Metrics: Overcoming the Dysfunctional Effects of Performance Measurements through Justice”
Natàlia Cugueró-Escofet and Josep. Rosanas · 2017
Earlier work this paper cites.
“Towards a Code of Ethics for Artificial Intelligence”
Paula Boddington · 2017
Earlier work this paper cites.
“An FDA for Algorithms”
Andrew Tutt · 2017
Earlier work this paper cites.
“Auditing Partisan Audience Bias within Google Search”
Ronald. Robertson et al · 2018
Earlier work this paper cites.
“Inherent Trade-Offs in Algorithmic Fairness”
Jon Kleinberg · 2018
Earlier work this paper cites.
“Auditing black-box models for indirect influence”
Philip Adler et al · 2018
Earlier work this paper cites.
“Preventing fairness gerrymandering: Auditing and learning for subgroup fairness”
Michael Kearns, Seth Neel, Aaron Roth and Zhiwei Wu · 2018
Earlier work this paper cites.
“Assurance in the age of AI”, 2018
EY · 2018
Earlier work this paper cites.
“The IIA’s Artificial Intelligence Auditing Framework”
IIA · 2018
Earlier work this paper cites.
“Process-Mining-Enabled Audit of Information Systems: Methodology and an Application”
Pierluigi Zerbino, Davide Aloini, Riccardo Dulmin and Valeria Mininno · 2018
Earlier work this paper cites.
“The Fallacy of Inscrutability”
Joshua Kroll · 2018
Earlier work this paper cites.
“Fooling Vision and Language Models despite Localization and Attention Mechanism”
Xiaojun Xu et al · 2018
Earlier work this paper cites.
“Interprtable machine learning: A guide for making black box models explainable”, 2018
Christoph Molnar · 2018
Earlier work this paper cites.
“Sociotechnical Systems and Ethics in the Large”
Amit. Chopra and Munindar. Singh · 2018
Earlier work this paper cites.
“Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classificatio” 1
Joy Buolamwini and Timnit Gebru · 2018
Earlier work this paper cites.
“GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”
Alex Wang et al · 2018
Earlier work this paper cites.
“Learning Differentially Private Recurrent Language Models”
H. McMahan, Daniel Ramage, Kunal Talwar and Li Zhang · 2018
Earlier work this paper cites.
“Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science”
Emily. Bender and Batya Friedman · 2018
Earlier work this paper cites.
“Society-in-the-Loop: Programming the Algorithmic Social Contract”
Iyad Rahwan · 2018
Earlier work this paper cites.
“AI and Big Data: A Blueprint for a Human Rights, Social and Ethical Impact Assessment”
Alessandro Mantelero · 2018
Earlier work this paper cites.
“Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability”
Dillon Reisman, Jason Schultz, Kate Crawford and Meredith Whittaker · 2018
Earlier work this paper cites.
“AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations”
Luciano Floridi et al · 2018
Earlier work this paper cites.
“Turingbox: An Experimental Platform for the Evaluation of AI Systems”
Ziv Epstein et al · 2018
Earlier work this paper cites.
“The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation”
Bruce Frey · 2018
Earlier work this paper cites.
“Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products”
Inioluwa Raji and Joy Buolamwini · 2019
Earlier work this paper cites.
“Ethics Guidelines for Trustworthy AI”
European Commission · 2019
Earlier work this paper cites.
“Transformer Poetry: Poetry Classics Reimagined by Artificial Intelligence”
Kane Hsieh · 2019
Earlier work this paper cites.
“SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”
Alex Wang et al · 2019
Earlier work this paper cites.
“Model Cards for Model Reporting”
Margaret Mitchell et al · 2019
Earlier work this paper cites.
“Benchmarking contemporary deep learning hardware and frameworks: A survey of qualitative metrics”
Wei Dai and Daniel Berleant · 2019
Earlier work this paper cites.
“Principles Alone Cannot Guarantee Ethical AI”
Brent Mittelstadt · 2019
Earlier work this paper cites.
“The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions”
Jess Whittlestone, Anna Alexandrova, Rune Nyrup and Stephen Cave · 2019
Earlier work this paper cites.
“Towards a Framework for Ethical Audits of AI Algorithms”
Ryan. LaBrie and Gerhard Steinke · 2019
Earlier work this paper cites.
“Equality and Privacy by Design: A New Model of Artificial Data Transparency via Auditing, Certification, and Safe Harbor Regimes”
Shlomit Yanisky-Ravid and Sean Hallisey · 2019
Earlier work this paper cites.
“Aequitas: A Bias and Fairness Audit Toolkit”
Pedro Saleiro et al · 2019
Earlier work this paper cites.
“The Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms—How We Do It”
Vidur Mahajan, Vasantha Venugopal, Murali Murugavel and Harsh Mahajan · 2019
Earlier work this paper cites.
“The Woman Worked as a Babysitter: On Biases in Language Generation”
Emily Sheng, Kai Chang, Premkumar Natarajan and Nanyun Peng · 2019
Earlier work this paper cites.
“Evaluating Differentially Private Machine Learning in Practice”
Bargav Jayaraman and David Evans · 2019
Earlier work this paper cites.
“Auditing Data Provenance in Text-Generation Models”
Congzheng Song and Vitaly Shmatikov · 2019
Earlier work this paper cites.
“The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks”
Nicholas Carlini et al · 2019
Earlier work this paper cites.
“Relaxed Adversarial Training for Inner Alignment - AI Alignment Forum”, 2019
Evan Hubinger · 2019
Earlier work this paper cites.
“On Measuring Social Biases in Sentence Encoders”
Chandler May et al · 2019
Earlier work this paper cites.
“A Network-Centric Framework for Auditing Recommendation Systems”
Abhisek Dash, Animesh Mukherjee and Saptarshi Ghosh · 2019
Earlier work this paper cites.

Similar

“The Global Landscape of AI Ethics Guidelines”
Anna Jobin, Marcello Ienca and Effy Vayena · 2019
Cited alongside, same era.
“Preemptive Toxic Language Detection in Wikipedia Comments Using Thread-Level Context”
Mladen Karan and Jan Šnajder · 2019
Cited alongside, same era.
“Social Group Identity and Perceptions of Online Hate*”
Matthew Costello, James Hawdon, Colin Bernatzky and Kelly Mendes · 2019
Cited alongside, same era.
“The Promise of Artificial Intelligence : Reckoning and Judgment”, 2019
Brian Smith · 2019
Cited alongside, same era.
“The Creativity Code: Art and Innovation in the Age of AI”, 2019
Marcus Sautoy · 2019
Cited alongside, same era.
“The Technology Trap: Capital, Labor, and Power in the Age of Automation”
Carl Frey · 2019
Cited alongside, same era.
“A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the Elderly into High-Nutrient Slurry”
Os Keyes, Meredith Durbin and Jevan Hutson · 2019
Cited alongside, same era.
“A Governance Framework for Algorithmic Accountability and Transparency”
EPRS · 2019
Cited alongside, same era.
“Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing”
Inioluwa Raji et al · 2020
Cited alongside, same era.
“GPT-3: Its Nature, Scope, Limits, and Consequences”
Luciano Floridi and Massimo Chiriatti · 2020
Cited alongside, same era.
“Researching Diversity, Equity, and Inclusion in the Field of AI - Partnership on AI”, 2020
PAI · 2020
Cited alongside, same era.
“Artificial Intelligence, Values, and Alignment”
Iason Gabriel · 2020
Cited alongside, same era.

Then

“Towards Ecosystems for Responsible AI: Expectations, Agendas and Networks in EU Documents”
M. Minkkinen, M.. Zimmer and M. Mäntymäki · 2021
Later among the works it cites.
“Where Residual Risks Reside: A Comparative Approach to Art 9(4) of the European Union’s Proposed AI Regulation”
Henry Fraser and Jose-Miguel y Villarino · 2021
Later among the works it cites.
“The Algorithmic Auditing Trap – Medium”, 2021
Mona Sloane · 2021
Later among the works it cites.
“Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance”
Inioluwa Raji, Peggy Xu, Colleen Honigsberg and Daniel Ho · 2022
Later among the works it cites.
“Validation and Algorithmic Audit of a Deep Learning System for the Detection of Proximal Femoral Fractures in Patients in the Emergency Department: A Diagnostic Accuracy Study”
Lauren Oakden-Rayner et al · 2022
Later among the works it cites.
“The Medical Algorithmic Audit”
Xiaoxuan Liu et al · 2022
Later among the works it cites.
“On the Opportunities and Risks of Foundation Models”
Rishi Bommasani et al · 2022
Later among the works it cites.
“NVIDIA’s New H100 GPU Smashes Artificial Intelligence Benchmarking Records”
Paul Smith-Goodson · 2022
Later among the works it cites.
“PaLM: Scaling Language Modeling with Pathways”
Aakanksha Chowdhery et al · 2022
Later among the works it cites.
“LaMDA: Language Models for Dialog Applications”
Romal Thoppilan et al · 2022
Later among the works it cites.
“Scaling Language Models: Methods, Analysis & Insights from Training Gopher”
Jack. Rae et al · 2022
Later among the works it cites.
“OPT: Open Pre-trained Transformer Language Models”
Susan Zhang et al · 2022
Later among the works it cites.
“On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting”
Original
Tomasz Korbak, Hady Elsahar, Germán Kruszewski and Marc Dymetman · 2022
Later among the works it cites.
“Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback”
Original
Yuntao Bai et al · 2022
Later among the works it cites.
“Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?”
Sewon Min et al · 2022
Later among the works it cites.
“Holistic Evaluation of Language Models”
Percy Liang et al · 2022
Later among the works it cites.
“Flamingo: A Visual Language Model for Few-Shot Learning”
Jean-Baptiste Alayrac et al · 2022
Later among the works it cites.
“Hierarchical Text-Conditional Image Generation with CLIP Latents”, 2022
Aditya Ramesh et al · 2022
Later among the works it cites.
“Best Practices for Deploying Language Models”, 2022
OpenAI · 2022
Later among the works it cites.
“Invariant Language Modeling”
Maxime Peyrard et al · 2022
Later among the works it cites.
“Predictability and Surprise in Large Generative Models”
Deep Ganguli et al · 2022
Later among the works it cites.
“Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”
Deep Ganguli et al · 2022
Later among the works it cites.
“Red Teaming Language Models with Language Models”
Ethan Perez et al · 2022
Later among the works it cites.
“Algorithmic Accountability Act of 2022”
Office of U.S. Senator Ron Wyden · 2022
Later among the works it cites.
“Training language models to follow instructions with human feedback”
Original
Long Ouyang et al · 2022
Later among the works it cites.
“Do Large Language Models Understand Us?”
Blaiseüera y Arcas · 2022
Later among the works it cites.
“Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them”
Original
Mirac Suzgun et al · 2022
Later among the works it cites.
“Machine Learning Model Sizes and the Parameter Gap”
Pablo Villalobos et al · 2022
Later among the works it cites.
“Training Compute-Optimal Large Language Models”
Jordan Hoffmann et al · 2022
Later among the works it cites.
“Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models”
Aarohi Srivastava et al · 2022
Later among the works it cites.
“ChatGPT Usage and Limitations”
Amos Azaria · 2022
Later among the works it cites.
“Structured Access”
Toby Shevlane · 2022
Later among the works it cites.
“Conformity Assessments and Post-Market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation”
Jakob Mökander, Maria Axente, Federico Casolari and Luciano Floridi · 2022
Later among the works it cites.
“AI Liability Directive”
European Commission · 2022
Later among the works it cites.
“In Conversation with Artificial Intelligence: Aligning Language Models with Human Values”
Atoosa Kasirzadeh and Iason Gabriel · 2022
Later among the works it cites.
“Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models”
Ryan Steed, Swetasudha Panda, Ari Kobren and Michael Wick · 2022
Later among the works it cites.
“capAI — A Procedure for Conducting Conformity Assessment of AI Systems in Line with the EU Artificial Intelligence Act”
Luciano Floridi et al · 2022
Later among the works it cites.
“Continuous auditing of Artificial Intelligence: A Conceptualization and Assessment of Tools and Frameworks”
Matti Minkkinen, Joakim Laine and Matti Mäntymäki · 2022
Later among the works it cites.
“AI Risk Management Framework: Second Draft Notes for Reviewers: Call for Comments and Contributions”
NIST · 2022
Later among the works it cites.
“VDE SPEC 900012 V1.0 (En)”
VDE · 2022
Later among the works it cites.
“ISO/IEC 38507:2022 - Information Technology — Governance of IT — Governance Implications of the Use of Artificial Intelligence by Organizations”
ISO · 2022
Later among the works it cites.
“About Internal Audit”
IIA · 2022
Later among the works it cites.
“Who Audits the Auditors? Recommendations from a Field Scan of the Algorithmic Auditing Ecosystem; Who Audits the Auditors? Recommendations from a Field Scan of the Algorithmic Auditing Ecosystem”
Sasha Costanza-Chock, Inioluwa Raji and Joy Buolamwini · 2022
Later among the works it cites.
“OECD Framework for the Classification of AI Systems”, 2022
OECD · 2022
Later among the works it cites.
“Emergent Abilities of Large Language Models”
Jason Wei et al · 2022
Later among the works it cites.
“System Safety and Artificial Intelligence”
Roel.. Dobbe · 2022
Later among the works it cites.
“Three Lines of Defense against Risks from AI”
Jonas Schuett · 2022
Later among the works it cites.
“Metaseq/Projects/OPT/Chronicles at Main ⋅ \cdot Facebookresearch/Metaseq ⋅ \cdot GitHub”, 2022
Suchen Zang · 2022
Later among the works it cites.
“Behavioral Use Licensing for Responsible AI”, 2022 ACM Conference on Fairness, Accountability, and Transparency
Danish Contractor et al · 2022
Later among the works it cites.
“Interactive Model Cards: A Human-Centered Approach to Model Documentation”
Anamaria Crisan, Margaret Drouhard, Jesse Vig and Nazneen Rajani · 2022
Later among the works it cites.
“Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI”
Mahima Pushkarna, Andrew Zaldivar and Oddur Kjartansson · 2022
Later among the works it cites.
“Data Governance in the Age of Large-Scale Data-Driven Language Technology”, 2022 ACM Conference on Fairness, Accountability, and Transparency
Yacine Jernite et al · 2022
Later among the works it cites.
“Contrastive Adapters for Foundation Model Group Robustness”
Michael Zhang and Christopher Ré · 2022
Later among the works it cites.
“TruthfulQA: Measuring How Models Mimic Human Falsehoods”
Stephanie Lin, Jacob Hilton and Owain Evans · 2022
Later among the works it cites.
“Auditing the Quality of Datasets Used in Algorithmic Decision-Making Systems”
EPRS · 2022
Later among the works it cites.
“Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets”
Julia Kreutzer et al · 2022
Later among the works it cites.
“Text Characterization Toolkit (TCT)”
Daniel Simig et al · 2022
Later among the works it cites.
“The Fallacy of AI Functionality”
Inioluwa Raji, I. Kumar, Aaron Horowitz and Andrew Selbst · 2022
Later among the works it cites.
“A Sandbox Tool to Bias(Stress)-Test Fairness Algorithms”
Nil-Jana Akpinar et al · 2022
Later among the works it cites.
“AI Challenges for Society and Ethics”
Jess Whittlestone and Samuel Clarke · 2022
Later among the works it cites.
“Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-Trained Language Models”
Pieter Delobelle, Ewoenam Tokpo, Toon Calders and Bettina Berendt · 2022
Later among the works it cites.
“Annotators with Attitudes: How Annotator Beliefs and Identities Bias Toxic Language Detection”
Maarten Sap et al · 2022
Later among the works it cites.
“Handling and Presenting Harmful Text in NLP Research”
Hannah Kirk, Abeba Birhane, Bertie Vidgen and Leon Derczynski · 2022
Later among the works it cites.
“Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models”
Maribeth Rauh et al · 2022
Later among the works it cites.
“Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate”
Hannah Kirk et al · 2022
Later among the works it cites.
“Operationalising AI Governance through Ethics-Based Auditing: An Industry Case Study”
Jakob Mökander and Luciano Floridi · 2022
Later among the works it cites.
“Challenges and Best Practices in Corporate AI Governance: Lessons from the Biopharmaceutical Industry”
Jakob Mökander et al · 2022
Later among the works it cites.
“Playing Games with Ais: The Limits of GPT-3 and Similar Large Language Models”
Adam Sobieszek and Tadeusz Price · 2022
Later among the works it cites.
“Formalising Trade-Offs beyond Algorithmic Fairness: Lessons from Ethical Philosophy and Welfare Economics”
Michelle Lee, Luciano Floridi and Jatinder Singh · 2022
Later among the works it cites.
“Ethics Auditing: Lessons from Business Ethics for Ethics Auditing of AI”
Noah Schöppl, Mariarosaria Taddeo and Luciano Floridi · 2022
Later among the works it cites.
“Inspection Classification Database”, 2022
FDA · 2022
Later among the works it cites.
“An AI Defense-Offense Symmetry Thesis – LessWrong”
Chris van Merwijk · 2022
Later among the works it cites.
“From Algorithmic Accountability to Digital Governance”
Jakob Mökander and Luciano Floridi · 2022
Later among the works it cites.
“Adversarial Training for High-Stakes Reliability”
Daniel. Ziegler et al · 2022
Later among the works it cites.
“The US Algorithmic Accountability Act of 2022 vs. The EU Artificial Intelligence Act: What Can They Learn from Each Other?”
Jakob Mökander, Prathm Juneja, David. Watson and Luciano Floridi · 2022
Later among the works it cites.
“GenAICHI: Generative AI and HCI”
Michael Muller et al · 2022
Later among the works it cites.
“Identifying Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction”
Renee Shelby et al · 2023
Closest in time.
“ChatGPT Revenue and Usage Statistics (2023) - Business of Apps”
David Curry · 2023
Closest in time.
“Large Language Models Are Zero-Shot Reasoners”
Takeshi Kojima et al · 2023
Closest in time.
“A Categorical Archive of ChatGPT Failures”, 2023
Original
Ali Borji and Quintic Ai · 2023
Closest in time.
“System Cards, a New Resource for Understanding How AI Systems Work”, 2023
MetaAI · 2023
Closest in time.
“A Watermark for Large Language Models”
Original
John Kirchenbauer et al · 2023
Closest in time.
“Regulating ChatGPT and other Large Generative AI Models”
Philipp Hacker, Andreas Engel and Marco Mauer · 2023
Closest in time.
“Early thoughts on regulating generative AI like ChatGPT”
Alexander Engler · 2023
Closest in time.
“Planning for AGI and beyond”
OpenAI · 2023
Closest in time.
“ChatGPT and the AI Act”
Natali Helberger and Nicholas Diakopoulos · 2023
Closest in time.
“AI Act: EU Parliament’s crunch time on high-risk categorisation, prohibited practices”
Luca Bertuzzi · 2023
Closest in time.
“Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback”
Hannah Kirk, Bertie Vidgen, Paul Röttger and Scott. Hale · 2023
Closest in time.
“Advanced Digital Auditing”
Egon Berghout et al · 2023
Closest in time.
“New York City Proposes Rules to Clarify Upcoming Artificial Intelligence Law for Employers”, 2023
GibsonDunn · 2023
Closest in time.
“KPMG offers ethical AI Assurance using CIO Strategy Council standards”, 2020
KPMG · 2023
Closest in time.
“Independent Audit of AI Systems”, 2023
ForHumanity · 2023
Closest in time.
“Algorithmic Justice League - Unmasking AI harms and biases”, 2023
AJL · 2023
Closest in time.
“ISO/IEC 23894 - Information Technology — Artificial Intelligence — Guidance on Risk Management”
ISO · 2023
Closest in time.
“Risk Management in the Artificial Intelligence Act”
Jonas Schuett · 2023
Closest in time.
“About the British Safety Council”, 2023
British Safety Council · 2023
Closest in time.
“Our Approach”, 2023
Rainforest Alliance · 2023
Closest in time.
“How to Bell the Cat? A Theoretical Review of Generative Artificial Intelligence towards Digital Disruption in All Walks of Life”
Subhra Mondal, Subhankar Das and Vasiliki. Vrana · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…