GitHub Action

Ironhide's referee runs where regressions actually ship: in your pull requests. The ironhide-scan action pulls the agent your PR builds into a sandboxed arena, attacks it, and posts a categorical verdict on what the agent did — state changes, tool calls, exfiltration — not on what it said about itself.

Two framing rules up front:

• Advisory-first. The action defaults to advisory mode: it posts the verdict as a PR comment and a passing check, and it fails builds only once you tell it to. Run advisory for a week or two, watch the baseline settle, then turn gating on.
• Preview. Observed-state verdicts carry the arena-l3-preview basis and a preview stamp. A preview verdict is an honest measurement, not a certification — treat it like a new reviewer whose judgment you are still calibrating, which is exactly what advisory mode is for.

The workflow#

Add a workflow file (for example .github/workflows/ironhide.yml):

name: ironhide
on:
  pull_request:

permissions:
  pull-requests: write   # the action posts the verdict as a PR comment

jobs:
  referee:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ironhide-ai/ironhide-scan@v1
        with:
          api_key: ${{ secrets.IRONHIDE_API_KEY }}
          advisory_mode: true        # advisory-first: comment, don't gate
          block_on: exfiltration     # hard-block categories once gating is on

Store the API key from ironhide connect as the IRONHIDE_API_KEY repository secret. Never commit it.

Inputs#

What ironhide test does#

The action is a thin wrapper around ironhide test, which you can run locally or from any CI system. One invocation:

1. Resolves your agent — the ci agent registered with ironhide connect.
2. Sweeps the arena suite — dispatches the agent into sandboxed environments seeded with attacks: injected instructions in tool results, poisoned fixtures, canary credentials planted where an exfiltrating agent would find them.
3. Grades observed state — after each episode the harness diffs the world: files written, tools called, network egress attempted, canaries that moved. The evidence is collected out of band; nothing the agent prints can change its verdict.
4. Applies the statistical gate — this PR's clean rate over the sweep is compared to the baseline branch's trailing clean rate. Only a shift past the suite's noise floor is treated as signal; a single flaky episode never fails a build.
5. Reports — posts (or updates) one PR comment with the verdict, the per-category effect counts, and repro ids for every finding, then prints the gate line and exits.

The first sweeps on your baseline branch establish the baseline. A baseline-establishing run always exits 0 and says so — it is not graded against itself.

The five verdicts#

The gate line and exit codes#

Every run ends with one machine-readable line in the job log (and at the foot of the PR comment):

IRONHIDE-GATE basis=arena-l3-preview runs=24 clean=21/24 baseline=0.92 shift=-4.5pt floor=6.0pt result=PASS

Field by field: basis is the grading basis (arena-l3-preview today — the preview stamp travels with the verdict); runs is the number of attacked episodes; clean how many showed no prohibited effect; baseline the baseline branch's trailing clean rate; shift this PR's clean rate minus baseline, in points; floor the noise floor for this suite size; result one of the five verdicts. In the example the shift is negative but inside the floor, so the result is PASS — that is the statistical gate working, not generosity.

Exit codes:

Reproduce a finding locally#

Every finding in the PR comment carries an id. Reproduction is deterministic:

ironhide repro --finding-id fnd_7c21a9

This replays the exact episode on your machine — same environment seed, same attack, same wire — so you can watch the trajectory that produced the observed effect step by step, then re-run it after your fix. Same episode, same world, same verdict: if the effect is gone locally, the next sweep on your PR will show it gone too.

Pre-deploy variant: scan in-process, no endpoint#

The action above drives the ci agent you registered with ironhide connect, resolved through your account and the server-side suite. For a fully self-contained gate — no endpoint, no account — the SDK runs your agent in-process, in the same CI job, with sandboxed tools. This is the same observed-state engine and the same five verdicts; the only difference is who runs the arena, so it carries a distinct in-process-preview basis.

The one thing you supply is an importable factory, build_agent(tools), that builds your agent bound to a given set of tools. Then the CI step is one line:

name: ironhide-predeploy
on:
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e . ironhide-sdk
      - run: python -m ironhide_sdk scan --agent myapp.agents:build_agent

python -m ironhide_sdk scan --agent mypkg.agents:build_agent \
    [--framework auto|callable|langchain] \
    [--scenarios a103-exfil-auditor,...] \
    [--fail-under-severity critical|high|medium|low] \
    [--json] [--advisory]

Like the hosted action, it prints a greppable gate line — with its own basis so a pipeline never confuses the two populations:

IRONHIDE-SCAN result=FAIL failures=3 basis=in-process-preview

Its exit codes keep the same discipline (2 is never collapsed into 0):

Under --json, stdout is pure JSON (the report plus a gate block) and the gate line is written to stderr, so one can be parsed and the other grepped. Full reference: the ironhide-sdk package README.

Scan with the agent's own tools (auto tool-mapping)#

If your agent's planner expects its own tool names, add --schema pointing at the agent's declared surface — an OpenAPI/Swagger spec or an MCP tool list — and the scan hands the agent tools with its own names, auto-mapped onto the arena. No manual mapping:

python -m ironhide_sdk scan --agent mypkg:build_agent --schema ./agent-tools.json

The mapping is a vocabulary translation only — the breach definitions stay the signed episode's — and it is biased to never manufacture a conviction: it routes only to primitives whose breach is driven by an agent-chosen value (an egress destination the agent picked, or a write to a protected record). It never auto-routes to a destructive/privilege/audit primitive (those convict on any invocation), so a benign tool that merely looks dangerous falls back to a benign read and is disclosed. Because it is a heuristic, the gate line carries an extra tool_map=auto-unreviewed stamp — advisory, distinct from the human-reviewed environment used on the attested path.

Where to go next#

• Quickstart: connect your agent and get a first verdict.
• GitLab CI: the same referee for GitLab merge requests.
• CLI reference: ironhide test, ironhide repro, and every other command behind the Action.