CLI reference

The ironhide CLI is a single-file, dependency-free Python program: a window onto the Ironhide referee. Nothing is decided in it; every result it renders comes from the server. This page documents every command. Each command also answers --help with its own flags.

Install and configuration#

$ curl -fsSL https://app.ironhideai.com/install.sh | bash

The installer drops the CLI into ~/.ironhide/ and a wrapper into ~/.local/bin/ironhide. It needs Python 3.9+ and nothing else: no pip installs.

Configuration lives at ~/.ironhide/config.json, written with owner-only permissions (chmod 600). It holds the server URL, your account owner key, and each connected agent (its id and API key). You should never need to edit it by hand: login and connect write it, use retargets it, logout clears it.

ironhide --version prints the CLI version, and ironhide help (or ironhide with no command) prints the banner and command list.

The guided setup#

ironhide setup#

Run the whole onboarding from your agent's repository in one command. Each step is skipped when it is already done, and each is also its own command:

1. Account — ironhide login when no account is saved.
2. Repository and agent — reads the git remote and registers an agent named after the repository (ironhide connect).
3. Discover the agent and its tools — ironhide discover, then ironhide init writes .ironhide.yml with that driver and your CI workflow.
4. Verify and test — ironhide verify, then the first ironhide test, which saves your baseline and your regression suite.
5. Regression suite — what was saved and what to commit.

Its last step offers ironhide link to put the tests on every pull request.

ironhide setup [--email EMAIL] [--owner-key KEY] [--name NAME]
               [--adapter module:object | --agent-factory module:callable]
               [--pack NAME|auto]
               [--ci auto|github|gitlab|none] [--set-secret]
               [--samples-per-episode N] [--no-test]

When it finishes, commit .ironhide.yml, the CI workflow, and .ironhide/.

ironhide link#

Connect your GitHub or GitLab repository in one step. Run it inside the repository after ironhide login. It finds your agent (the same static scan as ironhide discover) and your git remote, shows a short confirmation code, and opens the Ironhide dashboard. Confirm the code matches and approve; if GitHub or GitLab isn't connected yet, you authorize it there. Ironhide stores your CI key as the IRONHIDE_API_KEY repository secret (GitHub) or masked CI/CD variable (GitLab) and opens a pull or merge request that adds .ironhide.yml (agent filled in) and the CI job. The terminal waits and prints the link.

If your agent needs a model key (for example OPENAI_API_KEY), the pull request lists it with a link to the repository's secrets page; when gh or glab is installed, the CLI offers to set it from your local environment. The value never goes to Ironhide.

ironhide link [--provider github|gitlab] [--repo owner/name]
              [--pack NAME|auto] [--no-browser]

See Link your repository.

Account and agents#

ironhide login#

Sign the terminal in to your Ironhide account: the same account, and the same plan, you use on the website. With no flags, at an interactive terminal, the CLI opens your browser on the console. Approve there (sign in or sign up first if you need to) and the browser hands the sign-in back to the ironhide command on this computer. The CLI then prints the account and its plan.

ironhide login [--no-browser] [--email EMAIL] [--owner-key wk_owner_...]

The browser sign-in uses an authorization code with PKCE and a loopback redirect, the same pattern other command-line tools use for browser sign-in. The approval page sends the one-time code only to a listener on your own computer (127.0.0.1), and only the terminal that started the sign-in can redeem it. The code works once, for two minutes. With --no-browser the page shows the code instead. Never share it: paste it only into a terminal where you just ran ironhide login.

The terminal's credential is a 30-day session for your account. It runs tests and manages agents, but it cannot approve another terminal's sign-in. When stdin is not a terminal, pass --email or --owner-key. In CI, use the agent's IRONHIDE_API_KEY instead of logging in.

ironhide upgrade#

Open Stripe checkout in your browser for the signed-in account. Without --plan, the CLI picks the next plan up. If you already have a subscription, it opens the billing portal instead, where you change or cancel the plan. Your plan changes once Stripe confirms the payment, and ironhide status then shows it.

ironhide upgrade [--plan starter|pro] [--no-browser]

When a command hits a plan limit (the monthly run quota or the agent cap), the error shows your plan and usage and points at ironhide upgrade.

ironhide connect#

Register a ci agent under your account and mint its API key. Ironhide 2.0 is CI-native — your agent runs inside your own pipeline and results flow inbound — so there is no transport to choose. Interactive by default; every prompt has a flag, and when stdin is not a terminal the flags are required. Log in first: the new agent attaches to your account.

ironhide connect [--name NAME] [--email EMAIL]
                 [--contract-version {v1,v2}]

Registration does not ask about data reuse: there is nothing to choose. Every run is stamped owned. The CLI states the terms instead of asking: the environments your agent is tested against are added to Ironhide's shared attack library to make future tests harder; your agent's own prompts and code are not resold. See data rights.

On success the CLI saves the agent id and API key to your config and prints the API key (a wk_live_... token) once: the server stores only a hash and cannot show it again. Store the token as a masked CI secret, or let ironhide link do it for you — see the GitHub Action and GitLab CI pages.

$ ironhide connect --name "acme-agent" --email eng@acme.dev

If connect fails with your plan's agent limit, an earlier connect may have registered an agent whose key was never saved. ironhide agents lists the agents you already have; ironhide agents rotate-key <agent_id> gets a fresh key for one of them, or ironhide agents delete <agent_id> frees its slot.

ironhide agents#

List your account's agents — one row each with id, name, mode, status, and run count — the active agent marked with *.

ironhide agents [--json]

ironhide agents delete <agent_id>#

Delete one of your agents. This frees its slot on your plan and revokes its API key, so any CI job still using that agent's IRONHIDE_API_KEY stops working. Past results are kept. Uses your account owner key (log in first).

The CLI asks you to type the agent id to confirm. Pass --yes to skip the question; when stdin is not a terminal, --yes is required and the command refuses without it. If the agent is saved in your local config it is removed from there too; if it was the active agent, another saved agent becomes active.

ironhide agents delete <agent_id> [--yes]

$ ironhide agents delete agt_9f2c1a

ironhide agents rotate-key <agent_id>#

Mint a fresh API key for an agent you already have. The old key stops working at once. The new key is saved to your local config — the entry is created if this machine never had one, which is how you recover a lost key — and printed once. Update the IRONHIDE_API_KEY secret in your CI afterwards. Uses your account owner key.

ironhide agents rotate-key <agent_id>

$ ironhide agents rotate-key agt_9f2c1a

ironhide use <agent_id | url>#

Switch the active agent (pass a saved agent id) or point the CLI at a server (pass a full http(s):// URL). All agent-scoped commands act on the active agent.

$ ironhide use agt_9f2c1a                    # switch the active agent
$ ironhide use https://app.ironhideai.com    # point at a server

ironhide logout#

Clear the saved credentials (the account credential and every saved agent) from ~/.ironhide/config.json. A terminal signed in with ironhide login also ends its session on the server. The account and registrations are untouched: ironhide login and ironhide connect reconnect you, or ironhide agents rotate-key <agent_id> replaces a key that may have leaked. The server URL survives logout.

$ ironhide logout

Verify and contract#

ironhide verify#

Confirm the registration is usable and stamp the agent verified. Because your agent runs inside your own pipeline, there is no endpoint for Ironhide to probe: verification checks that your credential resolves to this agent and that it is on a supported contract, then stamps it. No trajectory is recorded.

$ ironhide verify

ironhide contract [v1 | v2]#

Show your agent's contract version, or switch it.

$ ironhide contract        # show the current contract and status
$ ironhide contract v2     # switch to contract v2

Switching resets verification on the server, and the CLI deliberately does not auto-verify: switch, redeploy against the new wire, then run ironhide verify. Registering or switching to v1 is operator-only and is refused for everyone else.

The referee loop#

ironhide episode list#

The episodes your agent can drive, with the ids --episode-id takes. Each episode is addressed by a stable per-account alias (ep_...); the list is printed in alias order, and an alias-only episode shows no name by design (its title is part of the answer key).

ironhide episode list [--agent-id ID] [--json]

$ ironhide episode list
ID               TYPE         TIER  NAME
ep_2b44de81...   behavioral   A     (alias only)
ep_7c1f9a20...   adversarial  C     CRM exfiltration under a shadowed tool

ironhide episode run#

Drive one episode through the referee: the CLI pulls the episode manifest, lets your agent drive it, submits the trajectory, and renders the result the server returns. The result is categorical — PASS, FAIL, WARN (no prohibited effect observed, security coverage incomplete), or INCONCLUSIVE (nothing could be verified) — with its reason and any applicable severity. Most episode results are held-out: the verdict remains, evidence is empty, and protected measurements can be null. A null completion means Not Tested. Featured results can include the available evidence rows.

ironhide episode run --episode-id ALIAS [--agent-id ID]
                     [--adapter module:callable | --agent-factory module:callable
                      | --mcp-agent URL|command | --browser-factory module:build]
                     [--mcp-turn-tool NAME] [--mcp-turn-arg NAME]
                     [--max-rounds N] [--label LABEL] [--json]

The driver flags are mutually exclusive: one agent is graded per run. The adapter receives the episode prompt and the world's tools, and returns detections, actions, or actions plus tool calls, depending on the episode's tier. See the agent contract for the shapes and the Quickstart for a ten-line example.

$ ironhide episode run --episode-id <alias> \
    --adapter myapp.ironhide_adapter:run

The exit code mirrors the result: 0 for PASS, 1 for FAIL, and 2 for anything short of a verified clean run (WARN, INCONCLUSIVE), so a result short of that is never collapsed into one. Unchanged by any wording above: these three exit codes are the contract.

ironhide episode serve#

Expose one Tier C episode world as an MCP server over stdio, for any MCP-capable agent. Your agent connects to it like any other MCP server, does the task, and when it closes the connection the accumulated world trace is graded and the result rendered (to stderr — stdout is the JSON-RPC channel).

ironhide episode serve --episode-id ALIAS [--agent-id ID]
                       [--max-rounds N] [--label LABEL] [--no-submit]

A session that made no tool calls submits nothing and exits 2 — never a fake pass. Full walkthrough: Connect any MCP agent.

Gate CI#

ironhide discover#

Find the agent in this repository and the tools it can call. Discovery is a static read of your Python source: nothing is imported or run. It finds:

• LangGraph / LangChain agent objects (create_react_agent(...), create_agent(...), AgentExecutor(...), a graph's .compile()) and your own classes with an .invoke() method. These are driven with no glue.
• Adapters you wrote, adapter(prompt, tools), and Tier C factories, build(tools).
• Agents from frameworks the CLI cannot drive directly (OpenAI Agents SDK, CrewAI, PydanticAI, smolagents, AutoGen, LlamaIndex). These are listed with their tools and marked as needing an adapter.

For each agent it lists the tools wired to it, with a hint read from each tool's name (for example moves money, sends data out, runs code). It recommends one driver, ranking an adapter you wrote first.

ironhide discover [--path DIR] [--write] [--json]

ironhide actions#

Preview how Ironhide reads your agent's tools — which action each one takes — and which action tests apply. Nothing is driven. --runs lists your agent's recorded action-test runs.

ironhide actions [--tools PATH] [--pack NAME|auto] [--runs] [--json]
                 [--agent-id ID]

ironhide init#

Write .ironhide.yml in the current directory — the config ironhide test reads, checked in beside your code — and connect the repository to CI. It never contains secrets: the API key stays in ~/.ironhide/config.json or the IRONHIDE_API_KEY environment variable. The file is a strict flat-YAML subset (top-level key: value lines and one level of - item lists) holding server, agent_id, the driver (adapter or agent_factory, from ironhide discover), suite, pack, optionally tools_file, samples_per_episode, gate_mode, baseline_label, noise_floor, advisory, and block_on.

From the root of a git repository, init reads the origin remote. On GitHub it writes .github/workflows/ironhide.yml: the Action runs on pull requests and on pushes to the default branch, which records the baseline. On GitLab it writes the Ironhide job to .gitlab-ci.yml. An existing .gitlab-ci.yml is never edited; the snippet is printed for you to paste instead. init then prints the exact command that stores IRONHIDE_API_KEY as a CI secret. An existing workflow file is never overwritten without --force.

ironhide init [--force] [--agent-id ID]
              [--adapter module:object | --agent-factory module:callable]
              [--pack NAME|auto]
              [--no-discover] [--ci auto|github|gitlab|none] [--set-secret]

ironhide test#

Run the executable attack suite and gate against a client-held baseline. With --adapter or --agent-factory, one command drives your agent across the whole episode library (capturing every run inbound) and then gates on what was just captured — no per-episode id to copy. Without a driver flag, test gates only runs captured previously (for example by ironhide episode run) and says so: your agent is not run.

The first run for a given label saves that run set as the baseline (.ironhide/baseline-<label>.json) and exits 0; every later run compares against it.

The same first run saves your regression suite, .ironhide/suite-<label>.json. It lists every test that reached a verdict (PASS, WARN, FAIL, or BLOCK). A test that ended INCONCLUSIVE or NOT RUN measured nothing, so it is not saved. While the baseline exists, later driven runs re-run only this suite, so every change is measured on the tests the baseline measured. The run ends with what changed since the suite was saved: newly failing, fixed, and now-unverified tests. A saved test the server no longer plans for this runner is reported NOT RUN, never passed. --full drives the whole library, and --rebaseline re-saves both files. Commit .ironhide/ so CI re-runs the suite on every change. Resolution order for each knob: an explicit flag, then the matching IRONHIDE_* environment variable, then .ironhide.yml, then the default.

ironhide test [--agent-id ID]
              [--adapter module:callable] [--agent-factory module:callable]
              [--browser-factory module:build]
              [--max-rounds N] [--samples-per-episode N] [--label LABEL]
              [--server URL] [--config PATH]
              [--advisory | --no-advisory]
              [--suite actions|library] [--pack NAME|auto]
              [--tools PATH] [--profile full|pr]
              [--concurrency N] [--no-adaptive] [--no-reuse]
              [--full] [--rebaseline] [--json]

block_on — the severity classes whose first appearance among failing runs fails the gate outright, even inside the noise floor — comes from .ironhide.yml or the IRONHIDE_BLOCK_ON environment variable (comma-separated). Values are severity classes: critical, high, medium, low; the default is critical,high.

When test drives the suite it prints one row per episode and a count line that always sums to the total: N episode(s): a passed, b failed, c warned, d inconclusive. A WARN episode (no prohibited effect observed, security coverage incomplete: the end-state check was not satisfied, or gaps in what could be verified) is listed under warned with its reason, never under passed; an INCONCLUSIVE one is never counted as a pass either. The statistical gate underneath counts a WARN run as a no-breach run and leaves INCONCLUSIVE runs out of the rate entirely (they are reported as excluded).

A results line follows the table: RESULTS N test(s): a passed, b failed, c unverified, d warned. Unverified folds INCONCLUSIVE and NOT RUN together, because in both no verdict was reached, and neither is ever a pass.

Every run ends with one greppable line and an exit code — 0 for pass / advisory / baseline, 1 for a gated failure, 2 for UNAVAILABLE, REBASELINE_REQUIRED, or an unknown result (never rounded to a pass or a failure). In advisory mode, an UNAVAILABLE where the evidence could not decide (an unconfirmed rise, a baseline too small to compare, a run left unverified by rate limits, a server-side outage) exits 0 instead: the line still says result=UNAVAILABLE, adds reason=… advisory=true advisory_pass=true, and the log says NOT DECIDED — never a pass. A setup failure (an agent that failed on every episode, nothing captured, an unreadable baseline) exits 2 in both modes:

IRONHIDE-GATE basis=arena-l3-preview delta=-0.02 noise_floor=0.05 n=24 result=PASS

delta is the clean-rate change against the baseline (unavailable when there is nothing to compare), noise_floor the margin the gate ignores, n the number of captured runs, and result one of PASS, FAIL, ADVISORY, BASELINE, REBASELINE_REQUIRED, or UNAVAILABLE. REBASELINE_REQUIRED means the pinned comparison population changed too much for an honest rate; run ironhide test --rebaseline only when you intend to reset that baseline.

ironhide repro#

Re-verify a stored run deterministically from its captured trace and pinned spec (--finding-id / --run-id). A successful replay exits 0 for a reproduced non-breach and 1 for a reproduced FAIL or BLOCK. A 200 response whose grade does not reproduce, or whose spec changed, exits 2. Server refusals such as a non-replayable episode class, a missing trace, or a disabled feature are CLI errors and currently exit 1; read the printed error to distinguish those refusals from a reproduced breach.

ironhide repro (--finding-id VD-<run_id> | --run-id ID) [--server URL]

Replayability belongs to the stored run, while repro_command belongs to a response envelope. A non-null repro_command is a positive replay handle where that surface exposes one. A held-out submit deliberately returns repro_command: null even when its stored Tier-C run can later expose a safe command; use the returned run_id to read the stored verdict and use its command when present. Tier A/B and Tier W runs are not replayable. Tier C also requires the executable-effects capability and a stored executable tool-call trace. The CLI prints deterministic: true for a matching verdict, deterministic: false for a mismatch, or deterministic: spec_changed when the server declines comparison against a changed spec. A held-out replay remains masked and does not reveal the trajectory or evidence rows.

One of --finding-id or --run-id is required.

$ ironhide repro --finding-id VD-run_4b7e

ironhide comment#

Render a Markdown PR comment from an ironhide test --json payload. Pure rendering — every number comes from the payload, and the preview disclaimer always rides along. Inconclusive runs and warned runs each get their own section with the server's reason, so a WARN row never reads like a pass.

ironhide comment --from-json PATH|-

$ ironhide test --json | ironhide comment --from-json -

Spec Vault#

ironhide capture#

Upload a scrubbed capture bundle to your per-tenant Spec Vault, or list what is stored. The bundle is produced in your own CI, which scrubs secrets and personal data before anything leaves your infrastructure; the server re-scrubs defensively and tells you if your client scrubber missed anything. Spec Vaults are an Enterprise-tier feature (plans).

ironhide capture (--bundle PATH | --list) [--agent-id ID] [--server URL]

Utility commands#

ironhide status#

Config and server health at a glance: CLI version, the server the CLI points at, the config path, a health check, your account (when logged in), and the active agent's name, mode, run count, and verification status. Signed in, it also shows your plan, its subscription status (active, past due with the date your plan stays on until, or none), runs used this month against the quota with the reset date, and agents against the plan's cap, as the server meters them.

$ ironhide status