Test the attacker who gets to try again

Security evaluation often gives the defender one clean chance. An attack is placed in front of a system. The system resists or fails. The result enters a table as if the encounter were final. Real attackers rarely accept that ceremony. They vary wording, timing, context, and delivery. When the target is probabilistic, they may simply repeat the same attempt until the system produces the rare behavior they need.

NIST’s work on agent hijacking evaluations demonstrated the difference. In one experiment, repeating injection attempts changed the average attack success rate across selected tasks from 57 percent on a single attempt to 80 percent over repeated attempts. The specific figures belong to that experiment. The durable lesson is that opportunity is a variable in the threat model.

This matters because agents consume adversarially writable material—email, web pages, documents, repository content—and can turn model output into action. A defense may reject malicious instructions most of the time. If the attacker can cause the material to be processed repeatedly, ‘most’ may describe a delay rather than a barrier.

Probability accumulates into exposure

A single-attempt success rate answers a narrow question: what happens during one draw from the system’s behavior under this setup? An operator needs a different answer: what is the chance of at least one damaging action over the number of opportunities an attacker can realistically obtain? The gap grows with retries, parallel agents, scheduled runs, and repeated ingestion of the same poisoned source.

The calculation is not merely mathematical because attempts are not always independent. An agent may remember earlier content, alter its plan, or grant a later tool call more context. A defender may rate-limit, detect repetition, or rotate prompts. An attacker may learn from partial failures. Evaluation should reproduce those dynamics instead of multiplying a laboratory rate and calling the result realistic.

When the attacker controls the number of draws, average resistance can become eventual permission.

The risk also crosses organizational boundaries. One malicious instruction can be copied into a ticket, summarized into a knowledge base, and retrieved by several agents. What appears to be one payload becomes many opportunities. The institution needs lineage for hostile or untrusted content so that downstream use does not reset the security history to zero.

Retries initiated by the system count as opportunities too. An agent that automatically retries after an error may expose a protected action to new stochastic outputs. A fallback model may interpret the content differently. A queue may replay work after a timeout even though the first execution succeeded but its acknowledgment was lost. Security review must include these operational paths.

Evaluate the environment, not a screenshot

NIST’s 2026 large-scale agent red-teaming analysis emphasizes the risks created when agents process external sources such as websites, email, and code repositories. A meaningful test should therefore include real trust boundaries: which content can carry instructions, which tools are reachable, which data can leave, and which actions are reversible.

The test should preserve evidence from near-successes. An agent may refuse the final harmful action while revealing that it accepted part of the attacker’s premise, disclosed a tool name, or staged data for transfer. Binary pass/fail scoring discards the path that shows how close the system came and which control actually stopped it.

Defense evaluation should separate model behavior from environmental constraint. If the model follows the injection but a permission boundary blocks the action, the system may have prevented harm, but the model defense did not pass. If the model refuses but the tool would have allowed the action, the environment remains dangerously permissive. Both facts matter for remediation.

Time belongs in the setup. How many attempts can an outsider make before detection? How often will the agent revisit the source? How long does a malicious instruction persist in memory or retrieval? How quickly can permissions be revoked? A system safe for a five-minute test may be unsafe across a month of unattended operation.

Control the number of chances

The first operational response is to reduce opportunity. Deduplicate equivalent work. Mark and quarantine suspicious sources. Place hard limits on retries for consequential actions. Require renewed authority when a fallback changes the decision path. Make successful actions idempotent so acknowledgment failures do not produce duplicate effects.

The second is to reduce consequence. Agents reading untrusted material should not automatically hold credentials for irreversible tools. Sensitive actions should pass through deterministic policy checks that do not reinterpret the hostile text. High-impact calls should require an independent approval context that excludes the content being evaluated.

The NIST AI 800-5 response analysis reports broad agreement that familiar cybersecurity practices remain relevant but require adaptation for agents. Repeated-opportunity testing is one such adaptation. Rate limits, least privilege, segmentation, monitoring, and incident response still matter; the new task is applying them to systems whose behavior varies between nominally identical runs.

An assurance report should state the attempt budget it tested, who controlled it, and what happened across the sequence. A one-shot result may be useful for comparing defenses, but it should not be presented as the probability that a deployed system survives a determined adversary. The attacker does not owe the benchmark its stopping rule.

Probabilistic systems do not make security impossible. They make exposure accounting unavoidable. Test the first attempt, then test the tenth, the replay, the parallel path, and the poisoned artifact that returns next week. A defense becomes credible when it governs the attacker’s opportunities as carefully as it measures the model’s responses.

— Dispatches · Summit Cognitive