A benchmark needs a budget

An agent can often improve a benchmark score by trying harder. It can think longer, call more tools, retry malformed requests, switch models, ask for hints, open more documents, or run the same test until one attempt succeeds. The final answer may look identical to an answer produced on the first try. The amount of system effort behind it is not.

This makes resource limits part of the measurement. A result obtained with ten minutes and three tool calls is not directly comparable to one obtained with two hours, fifty calls, and a human correction. If the budget remains unstated, performance absorbs an unknown quantity of time, money, infrastructure, and assistance.

NIST's January 2026 draft, Practices for Automated Benchmark Evaluations of Language Models, covers AI agent systems and treats evaluation-protocol design as a distinct stage. That protocol is where a credible benchmark defines what an attempt may consume and when the attempt must end.

Every recovery spends something

Recovery is part of agent capability. A useful system should notice a tool error, correct its arguments, and continue. But a benchmark that permits recovery has to count it. Otherwise a system that repeatedly fails and repairs itself can appear equivalent to one that performs the operation correctly at once.

The relevant budget is multidimensional. Wall-clock time captures delay but not parallel computation. Token counts capture model use but not expensive external tools. Tool-call counts ignore the difference between a database lookup and a long-running simulation. Dollar cost is comparable in one sense but moves with vendor pricing and may conceal operational risk. A serious evaluation reports the dimensions that matter to the intended use.

Human assistance belongs in the budget too. A clarifying answer, a repaired environment, or a hint from the evaluator can be appropriate when the target system includes a human partner. It should not disappear into the harness. Record when assistance became available, what information it supplied, and whether every system received the same opportunity.

An agent score is performance under a budget, even when the benchmark forgets to state the budget.

Retries require particular care. Repeating a stochastic attempt raises the probability that at least one run succeeds. Reporting only the best run changes the question from “how reliable is this system?” to “can this system ever succeed if given enough chances?” Both questions can be useful. They are not the same claim.

The stopping rule defines failure

An agent episode is a sequence, so the evaluator must decide when to stop it. A terminal answer is an obvious boundary, but not always the right one. The agent may loop without producing an answer, continue after taking a harmful action, or declare completion while a delegated task remains active.

Stopping at the first harmful action measures prevention. Continuing to the end may measure recovery or total damage. Stopping at a timeout treats slowness as failure. Extending the timeout may reveal eventual capability at an operationally useless latency. The rule decides which of these properties the score represents.

The rule should be written before results are seen and applied consistently. Otherwise evaluators can rescue favored systems by extending time, discounting infrastructure failures, or rerunning ambiguous cases. A defensible protocol distinguishes evaluator errors from system errors and specifies how each affects the denominator.

Multi-agent systems make this harder. Does the parent's budget include all child-agent work? Can children run in parallel? Does a failed child consume the attempt even if the parent chooses another? Counting only the visible orchestrator creates an unlimited accounting perimeter: work moves out of view while the headline score remains cheap.

Benchmark portfolios such as those maintained by MLCommons distinguish performance, quality, and safety purposes. Agent evaluation needs the same discipline inside each run. Completion rate, latency, cost, harmful actions, and resource use are separate outcomes that should not be compressed into one flattering number without an explicit decision rule.

Report a frontier, not a magic point

A single fixed budget is useful for comparison, but it can hide the system's operating curve. Run meaningful budget levels and report how success, cost, latency, and failure types change. One system may be better for fast routine work while another improves sharply when given more time. The frontier tells an operator where each system is useful.

Budget accounting should be monotonic and auditable. A retry, delegation, or route change must not reset the meter merely because another component performs the work. Otherwise the system can appear efficient by moving expenditure across an internal boundary the evaluator chose not to count.

Production budgets should connect to the evaluation. If the deployed router allows fewer tokens, different retries, or slower tools, the benchmark claim does not transfer unchanged. If premium users receive a larger budget, the institution is allocating different chances of success. That may be legitimate, but it is a service-policy choice, not merely a performance optimization.

Monitoring can then test whether real episodes stay inside the evaluated region. Track attempts that exceed budget, loops terminated by safeguards, hidden human interventions, and fallbacks that reset counters. An apparently efficient agent may be exporting cost to people who repair its work or to downstream services that absorb repeated calls.

Qualified reporting is more useful than a bare score: this system completed this share of tasks within these time, call, cost, and assistance limits, under this stopping rule. That sentence can support a deployment decision. “The agent scored seventy-two” cannot say how much effort seventy-two required.

Capability is not independent of resources. Agent benchmarks should make that dependence visible. The budget is not a footnote to the result. It is one of the conditions that makes the result true.

— Dispatches · Summit Cognitive