An agent can be perfectly healthy at the infrastructure layer while being terrible at its job. A 200 response does not mean the agent chose the right tool. A fast response does not mean the answer was grounded. Observability needs to span infrastructure, agent behavior, and business outcomes.
The goal is not to collect everything. The goal is to collect enough evidence to explain why a user-visible result happened.
Three levels of observability
- Infrastructure. Latency, TTFT, 5xx rates, timeouts, queue depth, rate limits, tool latency, streaming failures.
- Agent runtime. Model turns, tool selection, arguments, retrieval results, confirmation steps, retries, handoffs, trajectory length.
- Business outcome. Task success, time saved, adoption, repeat usage, escalation, correction rate, unsafe-action rate, cost per successful task.
Separate model quality from serving quality
A regression in user experience might come from a worse model decision or from a slower serving stack. Replay the same requests in a controlled environment and compare old/new model and old/new serving components separately. Keep model-quality metrics and infrastructure metrics distinct so you do not fix the wrong layer.
Failure taxonomy beats “agent accuracy”
Useful categories include wrong intent, wrong route, wrong tool, wrong arguments, retrieval miss, bad context, hallucination, policy misunderstanding, premature action, missing confirmation, backend failure, and serving failure. Once failures are categorized, remediation becomes targeted.
1production usage2 ↓3trace + outcome metrics4 ↓5failure taxonomy6 ↓7new regression cases8 ↓9targeted change10 ↓11offline eval12 ↓13canary / controlled releaseMeasure cost per successful task
Raw token cost can mislead. A cheap model that fails and causes a retry may cost more per successful task than a stronger model that finishes correctly. Define the quality bar first, then compare candidate models on success, safety, latency, and cost using the same eval set.
A 30-day launch review
After 30 days, ask: are people using the agent, are they completing real workflows faster, where are they correcting it, which tools fail, which routes are confused, what actions are being denied, how often do long-running tasks time out, and which failure modes are absent from the current eval suite? The strongest output of that review is a prioritized list of product changes plus a larger regression set.
PFPLabs takeaways
- Observe infrastructure, agent behavior, and business outcomes separately.
- Use traces as evidence, not just logs.
- Build a failure taxonomy before tuning prompts.
- Measure cost per successful task.
- Close the loop: production failures → eval cases → targeted fixes → controlled release. That is how the system compounds quality over time.