IRONHIDEWhy Ironhide
Transcripts can be talked around. The state of the world can't.
Software that acts gets tested on what it does. Agents somehow don't — they get graded on what they say. Ironhide is the pre-deploy referee for action-taking agents: it drives your real agent through multi-turn adversarial episodes in a sandboxed world, grades the state it left behind, and posts the verdict on the pull request.
Why a referee
Your agent's builder, your model vendor, and your own eval prompt all have a stake in a pass. A referee doesn't. Ironhide runs your real agent, records what happened to the world, and reports it — it has no incentive to call a breach a refusal.
Why observed-state
A transcript is what the agent says about itself. State is what actually happened: rows deleted, tools called, canaries that left the sandbox. Ironhide grades the diff of the world, so a confident apology never outranks a dropped table.
Why in CI
Agent behavior shifts when the prompt, the model, or a tool changes — which is every pull request. A quarterly review can't see that. A gate on the PR catches the regression in the same diff that caused it.
The comparison
How agent security gets bought today
Every row buys you confidence a different way. Only one of them grades what your agent actually did to a world under attack, before the merge, with a finding you can replay.
| Approach | Grades actions, not text | World under attack | Runs on every PR | Replayable finding | No production exposure |
|---|---|---|---|---|---|
LLM-as-judge eval A second model reads the transcript and scores it | ✕ | ✕ | ✓ | ✕ | ✓ |
Static injection benchmark Fixed public prompt set, text-only pass/fail | ✕ | ~ | ✓ | ✓ | ✓ |
Runtime guardrail Filters live traffic, after deploy, where mistakes are real | ~ | ✕ | ✕ | ✕ | ✕ |
Manual agent red team Humans probe a staging build, weeks per engagement | ✓ | ✓ | ✕ | ~ | ✓ |
Ironhide Sandboxed world, attacked; the state it leaves is the verdict | ✓ | ✓ | ✓ | ✓ | ✓ |
The others grade what your agent says, or catch it after deploy; Ironhide grades what it did, before the merge.
Not a pen test
A pen test probes your infrastructure, once, and hands you a PDF. Ironhide doesn't test your systems at all — it tests the agent you're about to hand them to, and re-tests it on every change.
Not a runtime guardrail
A guardrail argues with your agent in production, where a mistake is already live. Ironhide is pre-deploy: the damage happens in a sandboxed world, the trace is recorded, and the verdict lands before the merge.
A pre-deploy referee
Ironhide wires into CI, drives your real agent through a sandboxed world seeded with attacks — prompt injection, tool shadowing, canary exfiltration — and grades the state it leaves behind. The verdict — pass/fail, severity, why, and evidence — is a check on the PR.
Observed-state verdicts are preview, never a certification.
A verdict tells you what your agent did in a sandboxed world under a specific set of attacks. That is strong evidence, and we refuse to dress it up as more:
A FAIL is a fact
Every failure is a recorded state change — a deleted row, a canary that left the sandbox — with a one-line repro. Replay it before you believe it.
A PASS is bounded
A pass means no attack in this run landed — not that none ever will. The attack set grows, and the gate re-runs on every PR.
Never a certificate
Preview verdicts carry the preview label everywhere they appear. They are never sold as certification and never dressed up as a guarantee.
Preview means preview: observed-state results are reported with the label attached and are never benchmarked, sold, or trained on as anything else.