Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture

Understand

Large Language Models (LLMs) have enabled the emergence of LLM agents, systems capable of pursuing under-specified goals and adapting after deployment.

• Evaluating such agents is challenging because their behavior is open ended, probabilistic, and shaped by system-level interactions over time.
• Traditional evaluation methods, built around fixed benchmarks and static test suites, fail to capture emergent behaviors or support continuous adaptation across the lifecycle.
• To ground a more systematic approach, we conduct a multivocal literature review (MLR) synthesizing academic and industrial evaluation practices.