A successful answer can conceal unsafe tool activity, while a failed answer can arise from lost source evidence or an incorrect final calculation. Evaluation needs reference cases, component-level diagnostics and complete execution trajectories.
Choose the test by the failure it can expose
| Test scope | Evidence it should retain | Detailed reference |
|---|---|---|
| Reference cases | Expected evidence, actions and held-out examples | Golden datasets |
| Retrieval and context stages | Whether required facts survive each stage | Evidence survival and oracle tests |
| Tool execution | Arguments, authorization, effects and final state | Agent trajectories |
| Release decision | Per-slice quality, latency, errors and task economics | Metrics and release gates |
A component test isolates a fault; an end-to-end run checks whether the full workflow completes under the intended permissions. Version the source corpus and pipeline configuration so a changed score can be traced to a changed input or component.
In this chapter
- Golden datasets, hard negatives and contaminationCreate representative cases and preserve independent holdouts.
- Metrics and release gatesMeasure retrieval, groundedness, tool completion, latency and cost at their failure stages.
- Evidence survival, oracle tests and stage-level diagnosticsFind where required facts disappear and isolate retrieval from generation limitations.
- Agent trajectory evaluation, tool safety and LLM judgesPolicy invariants, judgment calibration and regression datasets.