Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems

Benchmarking of Multiagent Systems

Anja Zöller, Franz Rothlauf, Torsten O Paulussen, and Armin Heinzl. 2004 · 2004

Earlier work this paper cites.

Web analytics overview

Guangzhi Zheng and Svetlana Peltsverger. 2015 · 2015

Earlier work this paper cites.

Where and why is my bot failing? A visual analytics approach for investigating failures in Chatbot conversation flows. In 2021 IEEE Visualization Conference (VIS) . IEEE, IEEE, Louisiana, USA, 141–145

Avi Yaeli and Sergey Zeltyn. 2021 · 2021

Earlier work this paper cites.

Understanding user-web interactions via web analytics

Bernard J Jansen. 2022 · 2022

Earlier work this paper cites.

Practical OpenTelemetry: Adopting Open Observability Standards Across Your Organization

Daniel Gomez Blanco. 2023 · 2023

Earlier work this paper cites.

Exploring the Sensitivity of LLMs’ Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters

Original

Manikanta Loya, Divya Anand Sinha, and Richard Futrell. 2023 · 2023

Earlier work this paper cites.

React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR) . Kigali, Rwanda

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023 · 2023

Earlier work this paper cites.

A Taxonomy of AgentOps for Enabling Observability of Foundation Model based Agents

Original

Liming Dong, Qinghua Lu, and Liming Zhu. 2024 · 2024

Cited alongside, same era.

Does Prompt Formatting Have Any Impact on LLM Performance?

Original

Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024 · 2024

Cited alongside, same era.

Mixtral of experts

Original

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al · 2024

Cited alongside, same era.

Harnessing Large Language Models (LLMs) Optimizing Performance, Monitoring, and Compliance

Edwin Jose and Prasad Prabhakaran. 2024 · 2024

Cited alongside, same era.

GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers

An investigation of prompt variations for zero-shot llm-based rankers

Original

Shuoqi Sun, Shengyao Zhuang, Shuai Wang, and Guido Zuccon. 2024 · 2024

Later among the works it cites.

LangSmith

LangChain. 2025 · 2025

Closest in time.

LangFuse

LangFuse. 2025 · 2025

Closest in time.

LangGraph

LangGraph. 2025 · 2025

Closest in time.

LangTrace

LangTrace. 2025 · 2025

Closest in time.

OpenLLMetry

Traceloop. 2025 · 2025

Closest in time.