Fetching the paper…

Reading the bibliography…

2024

AgentOps: Enabling Observability of LLM Agents

Dong, Liming, Lu, Qinghua, Zhu, Liming

Original
PDF
Post on X

Understand

Large language model (LLM) agents have demonstrated remarkable capabilities across various domains, gaining extensive attention from academia and industry.

However, these agents raise significant concerns on AI safety due to their autonomous and non-deterministic behavior, as well as continuous evolving nature .
From a DevOps perspective, enabling observability in agents is necessary to ensuring AI safety, as stakeholders can gain insights into the agents' inner workings, allowing them to proactively understand the agents, detect anomalies, and prevent potential failures.
Therefore, in this paper, we present a comprehensive taxonomy of AgentOps, identifying the artifacts and associated data that should be traced throughout the entire lifecycle of agents to achieve effective observability.

AgentOps: Enabling Observability of LLM Agents · Around

Built on

B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering version 2.3,” Software Engineering Group, School of Computer Science and Mathematics, Keele University and Department of Computer Science University of Durham, Tech. Rep., 2007
2007
Earlier work this paper cites.
V. Garousi, M. Felderer, and M. V. Mäntylä, “Guidelines for including grey literature and conducting multivocal literature reviews in software engineering,” Inf. Softw. Technol. , vol. 106, pp. 101–121, 2019. [Online]. Available: https://doi.org/10.1016/j.infsof.2018.09.006
2018
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258 , 2021
Original
2021
Earlier work this paper cites.
Q. Lu, L. Zhu, J. Whittle, X. Xu et al. , Responsible AI: Best Practices for Creating Trustworthy AI Systems . Addison-Wesley, 2023
2023
Earlier work this paper cites.

Similar

Y. Liu, S. K. Lo, Q. Lu, L. Zhu, D. Zhao, X. Xu, S. Harrer, and J. Whittle, “Agent design pattern catalogue: A collection of architectural patterns for foundation model based agents,” 2024. [Online]. Available: https://arxiv.org/abs/2405.10467
Original
2024
Cited alongside, same era.
Q. Lu, L. Zhu, X. Xu, Z. Xing, S. Harrer, and J. Whittle, “Towards responsible generative ai: A reference architecture for designing foundation model based agents,” in 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C) . IEEE, 2024, pp. 119–126
2024
Cited alongside, same era.
B. Xia, Q. Lu, L. Zhu, Z. Xing, D. Zhao, and H. Zhang, “An evaluation-driven approach to designing llm agents: Process and architecture,” 2024. [Online]. Available: https://arxiv.org/abs/2411.13768
Original
2024
Cited alongside, same era.
M. Shamsujjoha, Q. Lu, D. Zhao, and L. Zhu, “Designing multi-layered runtime guardrails for foundation model based agents: Swiss cheese model for ai safety by design,” 2024. [Online]. Available: https://arxiv.org/abs/2408.02205
Original

Then

S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y. Li, A. Gupta, H. Han, S. Schulhoff, P. S. Dulepet, S. Vidyadhara, D. Ki, S. Agrawal, C. Pham, G. Kroiz, F. Li, H. Tao, A. Srivastava, H. D. Costa, S. Gupta, M. L. Rogers, I. Goncearenco, G. Sarli, I. Galynker, D. Peskoff, M. Carpuat, J. White, S. Anadkat, A. Hoyle, and P. Resnik, “The prompt report: A systematic survey of prompting techniques,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06608
Original
2024
Closest in time.
A. Chan, C. Ezell, M. Kaufmann, K. Wei, L. Hammond, H. Bradley, E. Bluemke, N. Rajkumar, D. Krueger, N. Kolt et al. , “Visibility into ai agents,” in The 2024 ACM Conference on Fairness, Accountability, and Transparency , 2024, pp. 958–973
2024
Closest in time.
L. Bass, Q. Lu, I. Weber, and L. Zhu, Engineering AI Systems: Architecture and DevOps Essentials . Addison-Wesley, 2025
2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2024

Cited alongside, same era.