Evaluate the route, not only the model

A model router promises a practical bargain. Easy work goes to a fast or inexpensive model. Difficult work goes to a stronger one. Sensitive work follows a constrained path. When capacity fails, a fallback keeps the service available. Each model can be evaluated separately, and the router can be tested for latency and cost.

None of those tests establishes that the assembled route is good. The system's consequential error may occur before inference, when the router misclassifies a hard case as easy, or after inference, when a timeout sends the request to a fallback that was never approved for that task. Component quality does not add up automatically to route quality.

This is a current evaluation problem, not an abstract one. NIST's 2026 benchmark practices emphasize defining the measurement target and evaluation protocol before reporting a claim. For routed systems, the target must include the routing policy and its runtime conditions.

The confusion matrix begins upstream

A router usually predicts a class: task type, difficulty, risk, or required capability. That prediction deserves its own error analysis. False simplicity sends hard cases down a weak path. False difficulty spends more time and money than needed. A false safety classification can expose data or tools to a model route that policy meant to exclude.

The most important examples lie near thresholds. A small wording change can alter a classifier score and send two equivalent requests to different models. An adversarial request can mimic a routine task. A novel case can receive high confidence because it resembles the wrong known class. Evaluating only the final answer hides these upstream errors whenever the selected model manages to recover.

Route evaluation should preserve both the selected path and plausible alternatives. This does not require running every request through every model in production. A sampled shadow evaluation can compare routes without executing their external actions. The purpose is to learn whether the router is assigning scrutiny consistently and whether its confidence predicts route regret.

The router fails whenever it assigns the wrong kind of judgment, even if the chosen model happens to answer well.

That distinction changes incident analysis. A bad output may reflect model failure, classifier failure, stale policy, unavailable capacity, or fallback behavior. If the record contains only the model that ultimately answered, every failure is blamed on the last component in the chain.

Fallbacks need their own test population

Fallbacks are often evaluated as availability mechanisms. Does the system keep responding when a provider times out? That is necessary, but incomplete. The fallback population is not random. It may be concentrated during outages, load spikes, regional failures, long contexts, or unusually difficult prompts—the very conditions under which ordinary benchmark results transfer least well.

A fallback test should recreate the reason the route changed. Simulate rate limits, truncated contexts, tool unavailability, policy denials, and partial responses. Measure not only completion but changes in error type, refusal behavior, tool selection, cost, latency, and need for human review. A route that preserves response rate while degrading authority boundaries has not preserved service.

OpenTelemetry's current GenAI conventions include operation names for agent invocation and tool execution, agent versions, and evaluation events in the evolving standard. The official GenAI conventions repository can help systems describe route observations consistently. Instrumentation makes evaluation possible at scale; it does not decide which route changes are acceptable.

The route record should include the policy version, classifier result and confidence, candidate models, selected model, material constraints, fallback trigger, and any override. Sensitive prompts need not be copied into every trace. Stable task features, digests, and controlled evidence references can support analysis without turning telemetry into an uncontrolled content store.

Make the deployment claim conditional

A route should be approved for a defined operating region: task classes, consequence levels, providers, model versions, latency budget, fallback set, and escalation rules. Changes outside that region should trigger targeted reevaluation. A new model is not a drop-in replacement merely because its average benchmark score is higher.

Cost needs to be treated as an outcome and a constraint, not a secret objective hidden inside the router. If budget pressure changes the route during an episode, record the budget state and the rule that acted on it. Otherwise a reviewer cannot distinguish a capability judgment from a spending limit. This is especially important when different customers, departments, or case classes receive different ceilings: an economic allocation can become an unequal allocation of scrutiny without appearing in quality metrics.

Abstention is part of route performance as well. A well-governed router may choose no model, defer until capacity returns, or require a human. Tests that count every non-answer as failure can pressure the system toward unsafe completion. Tests that ignore abstentions can make a cautious route look artificially accurate. Report them as a separate decision with its own timeliness and consequence.

The same is true of a policy threshold. Moving a cutoff can change which people or cases receive the stronger path without changing either model. The evaluation should report performance by route and by relevant case class, including abstentions and escalations. Aggregate quality can remain flat while one population loses scrutiny.

Operational monitoring should test the assumptions behind the route. Is the class distribution drifting? Are fallback rates rising? Are humans overturning one route more often? Are route decisions becoming less stable under equivalent inputs? Those are signals that the routing policy, not merely the model, needs review.

Model evaluation asks whether an instrument can perform. Route evaluation asks whether the institution chooses the right instrument under pressure. Agent systems need both. The second decides where the first is allowed to matter.

— Dispatches · Summit Cognitive