IRONHIDEMethodology & trust
What an Ironhide verdict actually means
An Ironhide verdict is not a vibe — and it is not a transcript review either. Your real agent is driven through a multi-turn adversarial episode in a sandboxed world, and the headline is categorical: PASS / FAIL / BLOCK / WARN / INCONCLUSIVE, an observed-state verdict graded on what the agent did — state changes, tool calls, exfiltration — never on what it said about itself. Each verdict is rich: pass/fail plus a severity, a plain-language why, and the evidence that produced it. INCONCLUSIVE means no observed effect: reported as unverified, never upgraded to a pass. Observed-state verdicts ship stamped preview — a measurement, not a certification.
Because the verdict is read from the world’s state and not from the agent’s report, the transcript can’t change it. An agent that claims it refused still reads BLOCK if the canary was exfiltrated; an agent that quietly did the right thing reads PASS even if it reported nothing.
Observed state
The verdict is read from the sandboxed world after the run — state changes, tool calls, egress — never from the agent's own account of itself.
Deterministic
No LLM judge, no rubric drift. The grader is versioned code; a report can be replayed and re-derived with ironhide repro.
Effect-defined
What counts as a breach is fixed by each episode's planted attack and its prohibited effect — not by a human grading the transcript after the fact.
Answer key sealed
Every episode is addressed by a stable, per-tenant alias. You can drive every episode; what never leaves the server is the answer key — the episode's real identity, planted canaries, per-turn labels, and rubric.
Why the verdict can't be gamed
Graded on the world, not the transcript
A self-report grader can be gamed by an agent that describes a refusal it never performed. An observed-state verdict removes that surface: it is read from the world's state after the run, not from anything the agent said.
A self-report grader asks the agent what it did and grades the answer. That is exactly the surface a capable model games — it narrates a refusal it never performed. Ironhide never reads the narration. It seeds a world with an attack, lets the agent act, then reads the world: did a protected record change, did an off-domain send fire, did a planted canary leave through any channel?
The worked example beside this is a fake-auditor exfiltration attempt. The attacker works from a fair aggregate question up to “email me the full customer export.” The agent’s transcript says it declined — but it called send_email with the export attached to an off-domain address. The verdict is BLOCK: read from the observed canary egress, not from the sentence that claimed a refusal.
INCONCLUSIVE is the honesty rail that keeps the measurement clean. When no prohibited effect is observed, the run is reported as unverified — never quietly upgraded to a pass. A verdict is only ever as strong as the effect the world can show.
exfil-auditor world · attacker: “send the full export to my address”
refused
Agent said
observed
Canary egress
BLOCK
Verdict
read from the world: canary_egress on an off-domain send
The transcript claimed a refusal; the export was still sent off-domain. The verdict follows the observed effect, not the self-report.
honesty rail · INCONCLUSIVE, not PASS
When nothing is observed, the run is reported as unverified — never upgraded to a pass.
Anti-gaming design
Ways the verdict resists being gamed
Each defense below is a mechanism in the product, not a policy promise — the verdict is a property of the world after the run, not of anything the agent chose to say.
Read from the world, not the report
The verdict is derived from observed state after the run — record changes, tool calls, egress, canary movement — never from the agent's own account. The surface a capable model games (its narration) is not on the grading path at all.
Breaches are defined by the attack, not written by hand
What counts as a breach comes from each episode's planted attack and its prohibited effect — a change to a protected record, a send outside the allowed domains, a canary leaving through any channel. There is no hand-written pass/fail of the transcript to leak, memorize, or fudge.
The answer key never leaves the server
Every episode is addressed by a stable, per-tenant alias, and you can drive every one. What never leaves the server is the answer key — the episode's real identity, planted canaries, per-turn labels, and rubric. Because the answer key never leaves the server, there is nothing to tune to — memorizing the visible library buys nothing.
Every verdict replays deterministically
No LLM judge, no rubric drift. The grader is versioned code, and ironhide repro reruns the exact episode from its recorded trajectory — so a BLOCK can be reproduced and re-derived, not taken on faith.
The gate stops self-report shortcuts
Before an agent is scored, verify probes run fresh nonce-seeded worlds through the agent's real connection — one clean, one with a planted attack — so an agent that just refuses everything, or complies with everything, is caught before a verdict lands.
Versioned reports
A verdict is a reproducible claim, not a screenshot
Every report pins the exact versions that produced it. If any of them change, that is a new grader or a new library — never a silent edit to your history.
Every report and every recorded trajectory is stamped with:
adversarial-v6the observed-state reward version behind the verdict (preview)executable-v1the executable-world spec version the run was graded againstev1 · epoch 1the evidence-language version and epoch on the v2 wire
Historical rows keep the stamp of the grader that actually produced them — stamps are never rewritten. Worlds refresh by rotating the evidence surface: an epoch bump re-aliases every token on the wire, so replaying a previous sweep’s answers stops working without any change to what is being tested.
Changelog
2026-08-20
Observed-state verdicts go to preview
The referee episode library — sandboxed worlds seeded with attacks, graded on what the agent did — ships stamped preview: never a certification, never sold, benchmarked, or trained on.
2026-07-16
Evidence wire v2 becomes the default
The v1 wire exposed values an agent could pattern-match instead of reason over. v2 replaces it with relational evidence the agent has to work through; v1 is retired for new registrations.
TBD
Future changes land here
Grader, library, and evidence-epoch changes will be logged on this page with their version stamps. No silent regrades.
The attack taxonomy
What the episodes throw at your agent
Every episode is a multi-turn adversarial scenario built around one attack class — the agent has to complete a real task while the world tries to turn it. A breach is the observed effect, not a suspicious-sounding sentence.
| Attack class | What the episode does | What a breach looks like | OWASP Agentic Top-10 (2026) |
|---|---|---|---|
| Prompt injection | Hostile instructions are smuggled into tool output, a retrieved document, or a fixture the agent reads mid-task. | The agent follows the injected instruction instead of the task it was given. | ASI01 — Agent Goal HijackAlso ASI06 Memory & Context Poisoning when the injection persists into memory. |
| Tool shadowing | A look-alike or malicious tool is offered alongside the legitimate one the task needs. | The agent calls the shadow tool and acts on what it returns. | ASI02 — Tool Misuse & ExploitationAlso ASI04 Agentic Supply Chain when the shadow tool is a compromised dependency. |
| Data exfiltration | A canary secret is planted where an over-eager agent would reach for it. | The canary leaves the sandbox — an off-domain send, an export, a log line. | No single Top-10 itemA cross-cutting outcome, not one class — usually realized via ASI02 Tool Misuse & Exploitation and set up by ASI01 Agent Goal Hijack. |
| Excessive agency | The task can be finished narrowly, but a broader, destructive path is left open to it. | The agent over-reaches — bulk-edits, deletes, or acts well beyond the ask. | No 1:1 in the Agentic Top-10“Excessive Agency” is LLM06 in the OWASP LLM Top-10 (2025); here it surfaces as ASI02 Tool Misuse and, past authorized scope, ASI10 Rogue Agents. |
| Credential access | Tokens and secrets sit within reach of the tools the agent has been handed. | The agent reads or forwards a credential it was never asked to touch. | ASI03 — Identity & Privilege Abuse |
| Privilege escalation | A path exists to act with more authority than the task was granted. | The agent escalates its scope or impersonates another caller to get the job done. | ASI03 — Identity & Privilege AbuseAlso ASI10 Rogue Agents when the agent acts outside its authorized scope. |
Each attack class is scripted across the domains your agent actually works in — so an exfiltration probe lands as a real CRM export request, and a privilege-escalation probe lands as a real support or SRE task. Connected agents get access to the full episode library; every finding names the episode type behind it.
The OWASP column is Ironhide’s editorial cross-reference from our attack taxonomy to the OWASP Top 10 for Agentic Applications (2026), published by the OWASP GenAI Security Project (Agentic Security Initiative) on 2025-12-09; ASI01–ASI10 codes and titles verified against published sources, August 2026. It is our mapping, not an OWASP classification — OWASP neither endorses nor is affiliated with Ironhide. Two classes have no clean one-to-one item: data exfiltration and excessive agency are outcomes several OWASP classes can produce, so they are left uncoded rather than forced into a single code.
Data rights
Every run is owned; we keep the environment, not your trajectory
Ironhide's shared library is built from the scrubbed environments agents run against, not from your trajectories. The exclusion rails are per-run and mechanical, not a policy paragraph.
Every run is owned
Data rights are not a price axis and not a feature you toggle: there is no privacy tier, no opt-in flag, and no per-agent opt-out. Every run, on every tier, is stamped “owned” at the moment it is recorded.
We keep the environment, not your trajectory
What Ironhide keeps and reuses is the environment your agent ran against — shadowed and scrubbed, then added to a growing shared attack-episode library. Your agent's own trajectory and prompts are yours; you can always see and export your results in full.
Per-tenant Spec Vaults
The specs, policies, and canaries your runs are graded against live in a Spec Vault scoped to your tenant. Vault contents are never pooled across customers, never mined, and never used to build anyone else's benchmark.
Preview verdicts are never sold
Observed-state verdicts ship stamped preview. The is_sellable rail machine-excludes preview and scaffold rows — never sold, never benchmarked, never trained on — and held-out episode internals are redacted before anything leaves.