Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Framework

A. B. Association et al. , “United states district court for the district of columbia, civil no. 95-1211 (crr); united states of america, plaintiff, v. american bar association, defendant: Report of the american bar association board of govenors,” American Bar Association, 1996

1996

Earlier work this paper cites.

P. M. Healy and K. G. Palepu, “The fall of enron,” Journal of economic perspectives , vol. 17, no. 2, pp. 3–26, 2003

2003

Earlier work this paper cites.

C. Coglianese, R. Zeckhauser, and E. Parson, “Seeking truth for power: Informational strategy and regulatory policymaking,” Minn. L. Rev. , vol. 89, p. 277, 2004

2004

Earlier work this paper cites.

M. C. Stephenson, “Information acquisition and institutional design,” Harvard Law Review , vol. 124, p. 1422, 2010

2010

Earlier work this paper cites.

F. R. Baumgartner and B. D. Jones, The politics of information: Problem definition and the course of public policy in America . University of Chicago Press, 2015

2015

Earlier work this paper cites.

J. C. Jung and E. Sharon, “The Volkswagen emissions scandal and its aftermath,” Global business and organizational excellence , vol. 38, no. 4, pp. 6–15, 2019

2019

Earlier work this paper cites.

B. Buchanan, J. Bansemer, D. Cary, J. Lucas, and M. Musser, “Automating cyber attacks,” Center for Security and Emerging Technology, Tech. Rep., November 2020. [Online]. Available: https://doi.org/10.51593/2020CA002

2020

Earlier work this paper cites.

M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O’Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas, M. Johnson, B. Laurie, A. Ingerman, I. Krawczuk, A. Askell, R. Cammarota, A. Lohn, D. Krueger, C. Stix, P. Henderson, L. Graham, C. Prunkl, B. Martin, E. Seger, N. Zilberman, S. Ó. hÉigeartaigh, F. Kroeger, G. Sastry, R. Kagan, A. Weller, B. Tse, E. Barnes, A. Dafoe, P. Scharre, A. Herbert-Voss, M. Rasser, S. Sodhani, C. Flynn, T. K. Gilbert, L. Dyer, S. Khan, Y. Bengio, and M. Anderljung, “Toward trustworthy AI development: Mechanisms for supporting verifiable claims,” arXiv preprint arxiv.org/abs/2004.07213 , Apr. 2020

Original

2020

Earlier work this paper cites.

S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealToxicityPrompts: Evaluating neural toxic degeneration in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 3356–3369. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.301

2020

Earlier work this paper cites.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

Original

2020

Earlier work this paper cites.

P. DeFazio and R. Larsen, “Final committee report—the design, development & certification of the Boeing 737 MAX,” The US House Committee on Transportation and Infrastructure, Subcommittee on Aviation. Washington, DC: US House Committee on Transportation and Infrastructure , 2020

2020

Earlier work this paper cites.

P. Cihon, J. Schuett, and S. D. Baum, “Corporate governance of artificial intelligence in the public interest,” Information , vol. 12, no. 7, p. 275, Jul. 2021

2021

Earlier work this paper cites.

D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia et al. , “Dynabench: Rethinking benchmarking in NLP,” arXiv preprint arXiv:2104.14337 , 2021

Original

2021

Earlier work this paper cites.

L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh, C. Biles, S. Brown, Z. Kenton, W. Hawkins, T. Stepleton, A. Birhane, L. A. Hendricks, L. Rimell, W. Isaac, J. Haas, S. Legassick, G. Irving, and I. Gabriel, “Taxonomy of risks posed by language models,” 2022 ACM Conference on Fairness, Accountability, and Transparency , jun 2022. [Online]. Available: https://doi.org/10.1145%2F3531146.3533088

2022

Earlier work this paper cites.

I. D. Raji, P. Xu, C. Honigsberg, and D. Ho, “Outsider oversight: Designing a third party audit ecosystem for AI governance,” in Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , 2022, pp. 557–571

2022

Earlier work this paper cites.

I. D. Raji, P. Xu, C. Honigsberg, and D. Ho, “Outsider oversight: Designing a third party audit ecosystem for AI governance,” in Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , 2022, pp. 557–571

2022

Earlier work this paper cites.

T. Shevlane, “Structured Access: An Emerging Paradigm for Safe AI Deployment,” in The Oxford Handbook of AI Governance . Oxford University Press, 2022

2022

Earlier work this paper cites.

P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al. , “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110 , 2022

Original

2022

Earlier work this paper cites.

I. Solaiman, Z. Talat, W. Agnew, L. Ahmad, D. Baker, S. L. Blodgett, H. Daumé III, J. Dodge, E. Evans, S. Hooker et al. , “Evaluating the social impact of generative AI systems in systems and society,” arXiv preprint arXiv:2306.05949 , 2023

Original

2023

Earlier work this paper cites.

J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova, “Generative language models and automated influence operations: Emerging threats and potential mitigations,” arXiv preprint arXiv:2301.04246 , 2023

Original

2023

Cited alongside, same era.

J. B. Sandbrink, “Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools,” arXiv preprint arXiv:2306.13952 , 2023

Original

2023

Cited alongside, same era.

E. Seger, A. Ovadya, D. Siddarth, B. Garfinkel, and A. Dafoe, “Democratising AI: Multiple meanings, goals, and methods,” in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , 2023, pp. 715–722

2023

Cited alongside, same era.

J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi, “Auditing large language models: a three-layered approach,” AI and Ethics , pp. 1–31, 2023

2023

Cited alongside, same era.

R. Burnell, W. Schellaert, J. Burden, T. D. Ullman, F. Martinez-Plumed, J. B. Tenenbaum, D. Rutar, L. G. Cheke, J. Sohl-Dickstein, M. Mitchell, and et al., “Rethink reporting of evaluation results in ai,” Science , vol. 380, no. 6641, p. 136–138, 2023

2023

Closest in time.

O’Brien, Ee, and Williams, “Deployment corrections: An incident response framework for frontier AI models,” Institute for AI Policy and Strategy, Tech. Rep., November 2023. [Online]. Available: https://www.iaps.ai/research/deployment-corrections

2023

Closest in time.

B. Bucknall and R. Trager, “Structured access for third-party research on frontier AI models,” Oxford Martin School, Tech. Rep., 2023. [Online]. Available: https://cdn.governance.ai/Structured_Access_for_Third-Party_Research.pdf

2023

Closest in time.

GOV.UK, “Safety and security risks of generative artificial intelligence to 2025 (annex b),” GOV.UK, Nov. 2023, available online. [Online]. Available: https://www.gov.uk/government/publications/frontier-ai-capabilities-and-risks-discussion-paper/safety-and-security-risks-of-generative-artificial-intelligence-to-2025-annex-b