The cleanest way to understand agent evals is to stop thinking of them as something inside the agent. The agent is the application under test. The eval system is the test harness around it. The grader is the assertion logic. The trace is the evidence of what happened.
That separation matters because expected answers and rubrics should not normally be visible to the production agent. Otherwise you contaminate the test.
1eval dataset2 ↓3test runner4 ↓5agent version under test6 ↓7sandboxed / replayed tools8 ↓9trace + output10 ↓11deterministic graders + model graders12 ↓13metrics / release gateThree layers of evaluation
- Outcome. Did the task actually succeed?
- Trajectory. Did the agent choose the right tools, arguments, order, and confirmation sequence?
- Behavior and safety. Did constraints hold even when the task was difficult or adversarial?
Deterministic graders first
If success can be asserted in code, assert it in code. Correct tool selected, exact order amount, no write before confirmation, correct customer scope, no forbidden destination—these do not need an LLM judge. Model graders are useful for fuzzy properties such as clarity, completeness, or whether a free-form answer is grounded.
A model judge also needs evaluation
An LLM-as-judge is not ground truth. Calibrate it against a human-labeled expert set. Measure agreement and look for biases toward verbosity, style, order, or related model families. Critical release gates should rely on deterministic checks or human-calibrated criteria where possible.
Mock destructive tools; replay external uncertainty
Offline agent evals should not issue real refunds, send real emails, or modify production CRM records. Use sandbox or fixtures for destructive actions. Replay external API responses when you want candidate versions to experience the exact same environment. Inject known failures such as 503s, timeouts, malformed responses, rate limits, and permission denials.
Version the whole agent
A useful eval record versions the model, prompt, tool schemas, retrieval configuration, orchestration logic, and agent configuration. Otherwise a “model comparison” can accidentally compare several changing variables at once.
Production failures become regression cases
The best eval set is alive. When production reveals a new misroute, prompt-injection attempt, bad retrieval result, or premature action, sanitize it and add it to the suite. Over time, the eval system becomes the product’s memory of what has gone wrong before.
PFPLabs takeaways
- Agent = app under test; eval = harness.
- Grade outcome, trajectory, and safety separately.
- Prefer deterministic graders where possible.
- Sandbox destructive tools. Do not turn evals into production incidents.
- Version everything and promote production failures into regression cases. The eval suite should evolve with reality.