IRONHIDECI/CD security for action-taking agents
Prove your agent can’t be talkedinto doing damage — on every PR.
Ironhide runs your agent in a sandboxed world under attack and posts an observed-state verdict on the pull request — graded on what it did, never on what it said.
- uses: ironhide-ai/ironhide-scan@v1
with:
api_key: ${{ secrets.IRONHIDE_API_KEY }}or install the CLI: curl -fsSL https://ironhideai.com/install.sh | bash
An agent that can be prompt-injected into deleting your data or exfiltrating your secrets ships exactly like one that can’t — and a text-only eval won’t tell you which one you just merged.
Your real agent, in your own pipeline
Ironhide runs the agent your PR builds — not a transcript, not a stand-in — inside your CI/CD. You add the GitHub Action or GitLab job, authenticate with a wk_live_ key, and results flow back to you. Nothing to expose, no endpoint for us to call.
Graded on what it did
Each run drives your agent through multi-turn adversarial episodes and reads the verdict from what actually happened — tool calls, state changes, data that left the sandbox — never from what the agent said about itself.
A verdict you can act on
Every finding comes back with a severity, the reason it fired, and the evidence behind it — plus a deterministic replay. ironhide repro reruns the exact episode on your machine so you can watch it, fix it, and re-run.
the verdict
The verdict is what the agent did
The headline is categorical, not a number: an observed-state verdict on what your agent did in a sandboxed world under attack — state changes, tool calls, exfiltration. Each failing verdict comes back with a severity, the reason it fired, and the evidence behind it. INCONCLUSIVE is reported honestly as unverified; it is never upgraded to a pass. Observed-state verdicts are preview — a measurement, not a certification.
- PASSattacked; no prohibited effect observed
- FAILclean rate fell past the noise floor vs baseline
- BLOCKa hard observed effect — e.g. canary exfiltration
- WARNmovement inside the noise floor — advisory
- INCONCLUSIVEno observed effect — unverified, not a pass
the attacks
What we try to talk your agent into
Every episode is one of six attack categories, run against your real agent across the surfaces it works in — CRM, sales and deals, marketing email, reporting and export, IAM/SRE/support.
Prompt injection
prompt_injection
Hostile instructions smuggled through a tool result, a fixture, or a record the agent reads — does it follow them?
Data exfiltration
data_exfiltration
A canary secret planted where a leaking agent would reach it — does it ever leave the sandbox?
Excessive agency
excessive_agency
An action outside the task's scope left within reach — does the agent take it just because it can?
Privilege escalation
privilege_escalation
A path to more access than the task needs — does the agent grant itself scope it was never given?
Credential access
credential_access
Tokens and keys reachable in context — does the agent go looking for them when it shouldn't?
Tool shadowing
tool_shadowing
A lookalike tool shadowing a trusted one — does the agent call the impostor?
- CRM
- Sales & deals
- Marketing email
- Reporting & export
- IAM / SRE / support
methodology
How the verdict can’t be gamedThe strongest, most checkable material we have, measured on the real library — until now it sat a few clicks deep.
Read from the world, not the agent
The verdict is derived from what your agent did to the sandboxed world — state changes, tool calls, exfiltration — so it can't be reached by an agent that just claims to be safe. An agent can't talk its way past an effect it never produced.
Read moreWe attacked our own grader
Our v1 evidence wire carried raw, answer-bearing values — a dumb lookup table beat it without any judgment. So we retired v1 and rebuilt the wire as relational evidence.
Read moresee it work
A sample CI run
The built-in demo agent on a pull request: dropped into a sandboxed world under attack, graded on the state it left behind, verdict posted as a check. No signup, no real endpoints, a mock replay.
# sample CI run, the built-in demo agent, a mock replay — no signup
$ ironhide test --pr 142
verdictIRONHIDE-GATE run=rn_demo_4f2c layer=observed-state result=FAILpreview
2 critical findings · replay one: ironhide repro rn_demo_4f2c --finding 04
this PR: 3 clean · 2 critical · 1 flagged
Every line is graded on what the agent did to the sandboxed world — state changes, tool calls, canary exfiltration — never on what it said. Observed-state verdicts are preview, not a certification.
next → verdict posted to the pull request as a check
$