Overview

Ironhide is a security referee for AI agents. You run your real agent, pre-production, inside your own CI/CD. Ironhide drives it through multi-turn adversarial episodes and returns a result on what your agent actually did, not on what it claims about itself. The result is categorical (pass / fail / warn / inconclusive) and carries its reason plus any applicable severity. Most episodes are held-out: the verdict remains, evidence is empty, and protected measurements can be null. Featured episodes can return available detail.

The one rule that governs everything: the referee reports observed behavior. When an episode plants an injected instruction, a poisoned tool result, or a canary credential, the harness watches what your agent does with it — the tool calls it makes, the state it changes, the data it moves. Nothing your agent prints about itself can change its result.

What Ironhide catches#

Ironhide runs your agent against episodes drawn from a taxonomy of real agent failure modes:

  • Prompt injection — instructions smuggled in through tool results, fixtures, or user content.
  • Tool shadowing — a malicious tool impersonating a trusted one.
  • Data exfiltration — sensitive data or credentials leaving where they should stay.
  • Excessive agency — an agent taking actions well beyond the task it was given.
  • Credential access — reaching for secrets it has no business touching.
  • Privilege escalation — acquiring or using access it was never granted.

Episodes span the workflows agents actually run: CRM, sales and deals, marketing email, reporting and export, and IAM / SRE / support. The verdict tells you whether the agent held and why. Applicable severity and observed detail are returned only when that response shape and visibility support them.

How a run works#

  1. Set up. ironhide setup, run in your agent's repository, signs you in, registers your agent, and finds the agent and the tools it can call. Your agent runs on your machine or in your CI; results flow inbound to Ironhide. There is no endpoint for you to expose and nothing Ironhide calls outbound.
  2. Attack it. The first ironhide test runs the action tests: planted instructions that try to make your agent take a harmful action with its own tools, in a test world where nothing real happens. (The episode library is also available: ironhide episode run drives one multi-turn episode at a time.)
  3. Get a result. The referee checks the observed trajectory and returns PASS, FAIL, WARN (no prohibited effect observed, security coverage incomplete), or INCONCLUSIVE (nothing could be verified), with a plain-English reason and any applicable severity. On a held-out episode, evidence is empty and protected measurements can be null; a null completion means Not Tested.
  4. Gate every pull request. ironhide link connects your GitHub or GitLab repository: it stores your key as a repository secret and opens a pull or merge request that adds the CI job. From then on each change re-runs your agent's saved tests and compares them with your baseline. See Link your repository.

The fastest path from an empty terminal to a first result is the Quickstart.

Advisory-first#

Every CI surface defaults to advisory mode: a would-be statistical FAIL is reported as ADVISORY and exits 0. A result the evidence could not decide (UNAVAILABLE) is reported as NOT DECIDED and, in advisory mode, does not fail the build either — it is never a pass. A setup failure you have to fix still exits 2 in both modes. Run advisory for a week or two, watch the baseline settle, then turn gating on. The measured runs do not change when you flip it — only whether a FAIL exits 1.

Once gating is on, a FAIL breaks the build: an observed prohibited effect, like a canary credential leaving the sandbox. By default the first appearance of a CRITICAL or HIGH breach fails the gate outright; anything else is judged statistically against your baseline.

Your data#

Every run is stamped owned — there is no privacy tier and no opt-out. What Ironhide keeps and reuses is the scrubbed environment your agent ran against, added to a growing shared attack library, not your agent's own trajectory or prompts. You can always see your own verdicts. Held-out answer-key detail and protected measurements remain masked in every customer export; featured results can include their available detail. Details on the data rights page.

Next steps#