The evaluator needs the failed trace

Evaluation reports prefer the surface that can be compared. Success rate, error rate, cost, latency, and judge score fit into a table. A trace does not. It contains false starts, tool calls, denials, retries, observations, and intermediate choices whose significance depends on the task. The table travels. The trace is retained briefly, restricted to a small team, or discarded after the aggregate is computed.

That economy is understandable for ordinary prediction. It is dangerous for agents. An agent can fail because it misunderstood the objective, selected the wrong tool, exceeded delegated authority, acted on stale data, recovered through luck, or completed the requested outcome by an unacceptable path. Those failures may receive the same zero. They do not describe the same system weakness.

NIST’s 2026 draft guidance on AI system evaluation emphasizes alignment among the evaluation objective, setting, and metric. For agent systems, the trace is what lets a reviewer test that alignment after the run. It connects the score to the behavior the evaluation was supposed to measure.

The same score can conceal opposite failures

Imagine two agents that fail to update a record. The first recognizes that the record is protected, asks for approval, receives no answer, and stops. The second writes to the wrong system and reports success because it never checks the result. Both may be marked incomplete. One exhibited appropriate restraint. The other revealed a control and verification failure. Improving the first means fixing workflow availability. Improving the second means changing agent behavior and authority.

Success has the same ambiguity. An agent might reach the correct final state after disclosing protected material to a search service, bypassing an approval through a different tool, or retrying until a nondeterministic safeguard yields. A grader that inspects only the final state calls the run successful. A security review calls the path the most important part of the result.

A score tells the evaluator where to look; the trace tells the evaluator what happened there.

The trace need not preserve every hidden internal detail. It needs the observable sequence required to reconstruct the task: prompts or bounded references to them, tool offers, model actions, environment observations, policy decisions, approvals, state changes, and the grader’s reasoning. Each event should carry stable identifiers and versions so a reviewer can distinguish the agent’s choice from the world it encountered.

This evidence also exposes evaluator errors. NIST AI 800-3 shows how evaluation assumptions shape performance measures and uncertainty; traces expose a complementary layer of assumptions inside the task and grader. A judge may reward a fluent explanation even though the environment shows no change. A state-based grader may mark failure after an external service applied the change with delay. A broken instruction may be impossible for any compliant agent to complete.

Failure families are more useful than failure counts

Once traces exist, evaluators can classify failure mechanisms instead of reporting a single remainder below the success bar. Planning errors, grounding errors, authorization violations, unsafe recovery, interface confusion, verification failures, and evaluator defects suggest different remedies. Their prevalence can change even when the overall score remains flat.

Classification should remain contestable. A trace may support more than one interpretation, and the label can inherit the evaluator’s bias. Good practice preserves the original evidence, the rubric version, the analyst or judge that assigned the class, confidence, and any later disagreement. The category is an aid to learning, not a replacement for the case.

Sampling can make review practical. High-consequence failures may deserve full inspection. Common low-consequence cases can be sampled by class. Successful runs should also be sampled because aggregate success can hide unacceptable paths. The sampling plan belongs in the evaluation report so readers know which behaviors received human examination and which were inferred from automated grading.

Privacy and security require deliberate retention. Traces may contain source data, credentials, proprietary instructions, or harmful material. The answer is not to discard the evidence by default. It is to minimize captured content, separate sensitive payloads from event metadata, restrict access, set justified retention windows, and preserve redacted forms that still support review.

Publish examples with the number

A trustworthy report should include representative traces or carefully redacted excerpts for the major success and failure classes. Readers can then see what the metric considers correct, how much assistance the agent received, what a denial looked like, and whether completion required recovery. Examples give the aggregate a behavioral meaning that a metric definition alone cannot supply.

Regression testing benefits too. When a control or model changes, teams can replay the situations that produced known failure mechanisms and compare the new path. A higher score with a new form of unsafe recovery should not be called unqualified improvement. Trace comparison shows whether the system actually learned the intended lesson or merely found another route to the grader’s target.

External evaluators need access to this layer under appropriate controls. An endpoint plus a final score may establish that someone independent ran requests. It does not let them inspect whether the deployed policy, tool use, and intermediate consequences matched the safety claim. Independence without sufficient observability can reproduce the institution’s blind spot under a different name.

Scores are necessary for scale, comparison, and trend. They should remain indexes into evidence rather than substitutes for it. Preserve the failed trace, sample the successful one, and keep the evaluator’s interpretation attached. An agent is a sequence of choices in an environment. Evaluation should retain enough of that sequence to know what the number counted.

— Dispatches · Summit Cognitive