Ironhide logoIRONHIDETest Your Agent

Overview

Ironhide is a security referee for AI agents. You run your real agent, pre-production, inside your own CI/CD. Ironhide drives it through multi-turn adversarial episodes and returns a verdict on what your agent actually did — not on what it claims about itself. The verdict is categorical (pass / fail), carries a severity, and comes with the why and the evidence behind it.

The one rule that governs everything: the referee grades observed behavior. When an episode plants an injected instruction, a poisoned tool result, or a canary credential, the harness watches what your agent does with it — the tool calls it makes, the state it changes, the data it moves. Nothing your agent prints about itself can change its verdict.

What Ironhide catches#

Ironhide runs your agent against episodes drawn from a taxonomy of real agent failure modes:

  • Prompt injection — instructions smuggled in through tool results, fixtures, or user content.
  • Tool shadowing — a malicious tool impersonating a trusted one.
  • Data exfiltration — sensitive data or credentials leaving where they should stay.
  • Excessive agency — an agent taking actions well beyond the task it was given.
  • Credential access — reaching for secrets it has no business touching.
  • Privilege escalation — acquiring or using access it was never granted.

Episodes span the workflows agents actually run: CRM, sales and deals, marketing email, reporting and export, and IAM / SRE / support. The verdict tells you which attack class was in play, whether your agent held, how bad a breach was, and exactly what it did.

How a run works#

  1. Connect. ironhide connect registers a ci agent under your account and mints an API key. Your agent runs in your CI; results flow inbound to Ironhide. There is no endpoint for you to expose and nothing Ironhide calls outbound.
  2. Drive an episode. ironhide episode run pulls one episode, hands the transcript and tools to a small adapter that wraps your real agent, and submits the trajectory.
  3. Get a verdict. The referee grades the observed trajectory and returns PASS / FAIL (or INCONCLUSIVE when there was nothing to grade), with a severity, a plain-English reason, and the evidence.
  4. Gate CI. ironhide test runs the executable attack suite and gates your pipeline against a client-held baseline — or drop in the GitHub Action or GitLab CI component and let the verdict surface as the ironhide/referee check on every pull or merge request.

The fastest path from an empty terminal to a first verdict is the Quickstart.

Advisory-first#

Every CI surface defaults to advisory mode: it posts the verdict and a passing check, and it fails builds only once you tell it to. Run advisory for a week or two, watch the baseline settle, then turn gating on. The verdicts don't change when you flip it — only whether a FAIL can break the build.

The gate is statistical: a run set's clean rate is compared to your baseline, and only a shift past the suite's noise floor is treated as signal, so a single flaky episode never fails a build. Hard-block categories (exfiltration by default) are the exception — one observed effect in a blocked category is enough, no statistics needed.

Your data#

Every run is stamped owned — there is no privacy tier and no opt-out. What Ironhide keeps and reuses is the scrubbed environment your agent ran against, added to a growing shared attack library, not your agent's own trajectory or prompts. You can always see and export your own results in full. Details on the data rights page.

Next steps#