A stochastic system can produce different trajectories on the same input. That means a single run is evidence, not a conclusion. The goal is to compare distributions of outcomes while spending evaluation budget where it is most useful.
Three ideas do most of the work: variance, ablation, and adaptive sampling.
Variance: how much the result moves around
Run the same case several times and you may see different tool paths, wording, or outcomes. That spread is variance. A 4% improvement between Agent A and Agent B is not meaningful until you understand whether normal run-to-run variance is 0.5% or 8%.
Ablation: isolate the responsible component
Ablation means removing or changing one component at a time to understand its effect. If a new agent regresses, hold prompt, tools, retrieval fixtures, and environment constant while changing only the model. Then repeat while changing only the prompt, only retrieval, or only a tool schema. It is controlled debugging for systems with many interacting components.
1same prompt + same tools + same fixtures + different model2same model + same tools + same fixtures + different prompt3same model + same prompt + same fixtures + retrieval disabledAdaptive sampling: spend runs where uncertainty is
You do not need twenty runs for every case. Start cheap across the whole suite, then allocate more repetitions to high-risk cases, high-variance cases, or comparisons where the difference between candidates remains ambiguous.
1Stage 1: cheap screen21–3 runs per case across full suite3 4Stage 2: focus55–10 runs on changed / high-risk / high-variance cases6 7Stage 3: release confidence8more repetitions only where the decision remains ambiguousKeep the environment controlled
When comparing A and B, use the same eval cases, retrieval fixtures, tool responses, and system configuration. Otherwise you cannot tell whether the outcome moved because the agent changed or because the world changed. After controlled comparison, test live robustness separately.
Gaming and overfitting
An eval score can improve for the wrong reason. An agent could learn to always escalate, which looks safe but destroys usefulness. Protect against that with holdout cases, rotating or procedurally generated cases, adversarial examples, production-derived cases, and multiple metrics rather than a single reward.
PFPLabs takeaways
- Variance = run-to-run spread.
- Ablation = change one thing to isolate cause.
- Adaptive sampling = spend more eval runs where risk or uncertainty is higher.
- Control fixtures before comparing versions.
- Do not optimize one metric until the agent games it. Safety without usefulness is still failure.