GitHub Action

Ironhide's referee runs where regressions actually ship: in your pull requests. The ironhide-scan action gates the runs your pipeline captured for the agent your PR builds — your agent driven through a sandboxed arena seeded with attacks — and posts a categorical result on what the agent did — state changes, tool calls, exfiltration — not on what it said about itself.

Two framing rules up front:

• Advisory-first. The action defaults to advisory mode: a would-be statistical failure is reported as ADVISORY and exits 0. When the evidence could not decide — a rise that could not be re-run to confirm it, a baseline too small to compare with, a run left unverified by your model's rate limits, or an outage on Ironhide's side (your plan's run quota is spent, or action tests are off on the server) — the result is UNAVAILABLE, the job log and PR comment say NOT DECIDED and why, and advisory mode does not fail the build for it. It is never a pass. A setup problem you have to fix (an agent that failed on every episode, nothing captured, a baseline that cannot be read) still exits 2 in advisory mode. Run advisory for a week or two, watch the baseline settle, then turn gating on.
• Preview. Observed-state results carry the arena-l3-preview basis and a preview stamp. A preview result is an honest measurement, not a certification — treat it like a new reviewer whose judgment you are still calibrating, which is exactly what advisory mode is for.

Verified installation#

Pin the Action to a release tag, not a moving alias — ironhide-ai/ironhide-scan@v2.5.0.

Set the repository variable IRONHIDE_CLI_SHA256 to the CLI SHA-256 published with that release. You usually don't have to: ironhide link sets it for you, and ironhide init --set-secret sets it alongside IRONHIDE_API_KEY (ironhide init without the flag prints the exact gh variable set command). It is a variable, not a secret — the value is public. The Action downloads the CLI artifact attached to the release and verifies its bytes before execution; a missing or mismatched digest fails the job, including in advisory mode. It never pipes an installer into a shell.

The artifact is immutable for the life of the tag, so the digest you pin stays valid — it does not change when the Ironhide server is updated. If you mirror the artifact internally, point cli_url at your copy and pin its digest; do not point it at a live server path, which would change underneath you.

Upgrading the Action is deliberate: move to the new tag and update IRONHIDE_CLI_SHA256 to the digest that release publishes. Never take a fresh digest from the same download in CI — that verifies a file against itself.

Fork pull requests and server requirements#

This Action executes the checked-out agent code with IRONHIDE_API_KEY in its environment. Do not give that secret to untrusted pull-request code. Fork pull requests normally do not receive repository secrets; an unavailable scan is not a passing security result. Do not switch to pull_request_target and check out or execute the untrusted head to bypass that restriction. Use a separately reviewed trusted revision and an isolated runner for any secret-bearing scan. See GitHub's guidance.

The Action reads the measured graded= count from the gate line. If no run was graded, or the count is missing, it fails closed with exit 2, even in advisory mode — unless the CLI itself declared an advisory pass (advisory_pass=true, below) for a can't-tell such as a server-side outage. An unavailable gate keeps its reported basis= token when present; it does not invent one if absent.

In .ironhide.yml, driver values use the CLI's simple unquoted key: value format. Quoted YAML values and inline comments are not supported; put comments on separate lines. ironhide init (and ironhide link) record the driver that ironhide discover finds; check that adapter or agent_factory names your agent before relying on this Action.

The workflow#

Add a workflow file (for example .github/workflows/ironhide.yml):

name: ironhide
on:
  pull_request:
  push:
    branches: [main]  # records the baseline for later PR jobs
  schedule:
    - cron: "17 3 * * *"  # nightly: every test, 5 samples, against the baseline
  workflow_dispatch:      # "Run workflow" by hand runs the same full pass

permissions:
  contents: read       # required by actions/checkout for private repositories
  pull-requests: write   # the action posts the result as a PR comment

jobs:
  referee:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ironhide-ai/ironhide-scan@v2.5.0
        with:
          api_key: ${{ secrets.IRONHIDE_API_KEY }}
          adapter: your.module:agent  # replace with your actual driver
          cli_sha256: ${{ vars.IRONHIDE_CLI_SHA256 }}
          advisory_mode: true        # advisory-first: comment, don't gate
          block_on: critical,high    # severity classes that fail the gate outright once gating is on

Store the API key from ironhide connect as the IRONHIDE_API_KEY repository secret. Never commit it.

The Action drives your agent directly with adapter or agent_factory. Install your project's dependencies before the Action and make the named module importable from the checkout. Alternatively commit the driver in .ironhide.yml:

agent_id: your-connected-agent-id
adapter: your.module:agent
samples_per_episode: 3

An explicit Action input replaces the project's driver selection. adapter and agent_factory are mutually exclusive. No configured driver fails the Action, even if old runs exist; zero eligible observations also fail. Advisory mode never converts these configuration or observation failures into success.

Inputs#

No input selects the run type: the CLI reads GITHUB_EVENT_NAME. A pull request runs one sample per action test and re-runs only what looks suspicious; a push to baseline_branch runs the full count; schedule and workflow_dispatch run every test 5 times (nightly_samples). See how a run is sampled.

The action exposes three outputs: result (the gate result token), gate_line (the full IRONHIDE-GATE line), and baseline_restored (true when the cache restore found a baseline).

What ironhide test does#

The action is a thin wrapper around ironhide test, which you can run locally or from any CI system. One invocation:

1. Resolves your agent — the ci agent registered with ironhide connect.
2. Drives the suite (only with --adapter or --agent-factory) — your agent is run through every episode in the library: sandboxed worlds seeded with attacks such as injected instructions in tool results, poisoned fixtures, and canary credentials planted where an exfiltrating agent would find them. Each run is submitted and graded as it happens. Without a driver flag this step is skipped and the runs captured earlier are used.
3. Checks observed state — the referee reads what each run did: records changed, tools called, egress attempted, canaries that moved. The evidence is collected out of band; nothing the agent prints can change its result.
4. Applies the gate — collects this agent's captured runs into a run set and compares it to the client-held baseline for the label (advisory until you turn gating on).
5. Reports — the CLI prints one row per verdict with status, applicable severity, episode alias, and a repro command only when replay is available, followed by the statistical sentence and gate line. The Action's PR comment is intentionally smaller: a result badge, an observed-state disclaimer, the IRONHIDE-GATE line, and generic repro guidance.

The first ironhide test for a label has nothing to compare against, so it saves that run set as the baseline (.ironhide/baseline-<label>.json, under the working directory) and exits 0 — it is not tested against itself. Keep that file between runs (for example with a cache step keyed on the label) or every run starts a fresh baseline.

The results#

Each captured run carries one of these results:

The gate itself answers with PASS, FAIL, ADVISORY (a would-be FAIL under advisory_mode: true), BASELINE (first run for the label), REBASELINE_REQUIRED (the pinned comparison population changed too much), or UNAVAILABLE (the gate could not be evaluated).

The gate line and exit codes#

Each completed measurement prints one machine-readable line in the job log (and at the foot of the PR comment):

IRONHIDE-GATE basis=arena-l3-preview delta=-0.02 noise_floor=0.05 n=24 result=PASS planned=24 graded=24

Field by field: basis is the preview basis (arena-l3-preview today — the preview stamp travels with the result); delta is this run set's clean rate minus the baseline's (unavailable when there is nothing to compare); noise_floor the margin the gate ignores; n the number of captured runs; result the gate outcome. In the example the delta is negative but inside the floor, so the result is PASS — that is the gate working, not generosity.

An UNAVAILABLE line also says why, and what advisory mode did with it:

IRONHIDE-GATE basis=statistical delta=+0 noise_floor=0.05 n=21 result=UNAVAILABLE graded=7 unverified=0 not_run=0 reason=unconfirmed advisory=true advisory_pass=true

reason is why the gate could not decide; advisory is the mode the CLI ran in; advisory_pass=true appears only on an UNAVAILABLE line, only in advisory mode, and only when the CLI exits 0 — the CLI decides it, so the Action needs no list of reasons:

Exit codes:

Reproduce a finding locally#

A replayable Tier-C row in the CLI output can show its complete repro command. The Action's compact PR comment does not include per-episode rows.

ironhide repro --finding-id VD-<run_id>

A non-null repro command is a positive handle on this stored-verdict surface. A held-out submit may have shown repro_command: null for the same underlying run. Tier A/B and Tier W runs are not replayable; Tier C also requires the executable-effects capability and a stored executable tool-call trace. The CLI prints deterministic: true for a matching verdict, deterministic: false for a mismatch, or deterministic: spec_changed when the server declines comparison against a changed spec. Most episodes are held-out, and their replayed payload stays masked: the status and applicable severity remain, while evidence is empty and protected measurements can be null. Replay confirms stability; it does not expose the held-out trajectory. Use a featured episode for step-by-step detail, then re-run your agent after the fix.

Either way, fix your agent and re-run it with ironhide episode run. If the effect is gone locally, the next sweep on your PR will show it gone too.

Where to go next#

• Quickstart: connect your agent and get a first result.
• Link your repository: ironhide link wires this workflow and the secret in one step.
• GitLab CI: the same tests and gate on GitLab merge requests.
• CLI reference: ironhide test, ironhide repro, and every other command behind the Action.